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Orientation and Overview 
Jack V. Edling 



The purpose of this manual is to communicate concepts which 
normally are presented by lecture. It contains the Informatlcn 
you are ifesponslble for In the course. But Information can be 
known at different degrees of proficiency. Therefore* after you 
have leazned the contents of the manual you will apply the con- 
cepts, where relevant, to various problems, these problems will 
be presented In "practice sessions" scheduled dally. Through this 
routine, ^'ou will become proficient with the concepts, using and 
manipulating them productively. 

However, only a limited time is available to cover knowledge 
that othexti have learned ^er many years of experience. It may not 
be possible, therefore, to gain the complete mastery that would 
be desirable. After you cosplete the practice session, you will 
take a new type of test (called a criterion test) to determine 
your mastery? of the concepts. If you demonstrate mastery, you 
will apply the concepts to your field and also get an evaluation 
of your creative work. If you do not exhibit mastery on the 
criterion test you will enter either indlvldtial or small group 
instruction. If time runs out you will be directed to biblio- 
graphical references to continue your studies. 

You will probably have questions about various phases of 
this program; for example, why was the Teaching Research Division 
selected to prepare and conduct this specialized research training 
Institute; or wtuit Is the nature of the concepts being taught; how 
were they selected; what Is my schedule as the institute la con- 
ducted? These and related questions will be answered in the 
orientation and overview. 



Orientation 

Teaching Research Division was organized by the Oregon State 
Board of Higher Education in I960. Its objective Is to improve 
the effectiveness and efficiency of Instruction without reference 
to grade level or subject matter area. One area of concern Is the 
Improvement of elementary and secondary education primarily 
through teaidier education programs. Also, it is concerned with 



lodustrial and nllitaty trainls^ in cooperation with tt^e Division 
of Continuing Education (established by the saiRe boardi) • But» Its 
primary mission Is to cooperate with the Institutions 1^ the 
Oregon State System of Higher Education » and higher Education 
Institutions over the country, to Improve the quality of uoder- 
greduate and gradi:ate instruction. ^ wmer 

In its seven years of existence the Division hae qomplated 
more than 80 rasaarch projects. The staff has gfot^-from 2 to more 
than 50 hl^Xy qualified researchers in the bohn^orqi sciences or 
support specialists essential for the cc^t^ulhg^ of the 
Dlvlelon. It has received nore than 40 grants to|4iing approxi- 
mately two million dollars. Each grant was ohtal&i4 hf ter the 
preparation of a proposal and a critical r^Leif hy l^er^^ 
advising either federal or privately supported f^d granting 
agencies. The operating budget for the current 'ilscai’ year is 
approximately three quarters of s million do^at®." 

The dlvlelon is engaged in three major classes of activity, 
^ch called a unit. The basic research unit ^s concerned with 
finding limitations in existing theories and practices. Although 
this aspect is negative In character, the basic research unit 
seeks specific evidence of whet le wrong In existing theories and 
practices and under what conditions. It then pushes for solutions 
to the gaps, the coritradictlons, and the limitations in existing 
knowledge. The contextual research unit, more positive in its 
approach, is applying what has been learned from research to the 
Improvonent of Instruction. Its concern is the development of 
more effective end efficient instructional materials, tests, pro- 
cedures and systems, combining them into the best possible 
learning experience for learners of all types. 

For the first six years of the Division’s history, a report 
or jounuil article ves prepared on coim>leted projects placed 
on a library shelf to gather dust except for use by occasional 
scholars. Hut a year ago, the Directorate decided that more 
active effort should be applied to disseminate the work of the 
Division, A dissemination unit was organized to conduct insti- 
tutes and employ other media to disseminate the findings of the 
Dlvlsjonf 



la the spring of 1967, the Division applied to the U.S. 
Comoisaiqper of Education for a grant to conduct a national 
research training Institute for participants In Consortium 
Research D^yelopment (CORD) Projects. A separate proposal was 
submitted the U.S. Commissioner of Education for a project to 




develop a program of materials for a short term educational 
research training program* This manual Is one result of that pro~ 
ject* 



Both the Teaching Research, Division and the U.S* Office of 
Bducatlon believe that this Research Training Program will be 
relevant to requirements of Institutions of hl^er education par- 
ticipating In CORD projects* The Teaching Research Division has 
as Its basic objective (and Its basic problem) production of 
change with minimum Involvement * Hlnlmum involvement Is essential 
because If Teaching Research were to continue to assume the major 
responsibility for the development of new Instructional systems 
It would be very limited In the number of departoients or agencies 
iw could serve* A more efficient way must be found to bring 
about change with less involvement by research and development 
specialists* The present attempt must be considered i&lnlmum 
Involvement* 

Our Intention is to spend a concentrated two weeks atteo^tlng 
to train you In our techniques with no opportunity for con- 
tinuing assistance* This may be below a realistic Involvement 
necessary to create change on your campuses* However > through 
the Consortium Program» you are combining your efforts In attempts 
to affect necessary changes* Whether this combination will be 
adequate Is unknown» for this effort Is an experiment* If you 
significantly improve your courses » then you wi.ll answer the 
question as to whether the present approach Is sufficient* 



Overview 

The ultimate objective of the Research Training Program Is 
to provide you with basic 8fci‘>le necessary to plan and produce 
an loproved Instructional system and to plan and conduct 
research related to Instruction* In order to achieve that 
ultimate objective It has been subdivided Into seven enabling 
objectives each of wbJ.ch contains concepts and skills essential 
to attainment of the ultimate objective* 

Throu^ experience it has been learned that when an Indi- 
vidual or department is given the charge, and the opport^^?sity, to 
Improve Its instructional procedures It normally begins by doing 
more of what It is already doing* Most professors believe they 
already know what they need to do to improve their courses, and 
all they need is sufficient time and resources to do what they 
have always wanted to' do — ani^ an approximation to the perfect 
course will result* Unfortunately, this approach has been tried 



extensively with the result that students are given the same con- 
tent and the same Ideas merely In a more elegant form* And the 
tragedy of this ls» that empirical evidence has consistently 
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sophisticated nature of the presentation* results In less learning 
by students. This is hard to accept* But* It Is a consistent 
finding* and It has a rather firm psychological basis. The fact 
Is that the requirements of the sophisticated learner* ax^ wl^t 
satisfies him* are very different from the requirements of the 
naive learner* and what satisfies him. The evidence Indicate^ 
that the more sophisticated and Informed the scholar* the less 
qxiallfled he Is to know the requirements of the naive learner! 



Therefore* to merely afford a scholar more opportunity to 
prepare a course of Instruction Is not a sufficient condition* In 
and of Itself, to Insure the Improvement of instruction* The 
steps In a process which has been demonstrated to be an effective 
means for developing more effective leanrlpg experiences are 
outlined in Figure 1* 



The first task In the development of an Improved Instructional 
system Is to Identify criteria which will be used to determine 
when an instructional objective has been attained* If there Is 
no identification, in measurable terms* of what the learner Is 
supposed to learn as a result of exposure to the Instructlcr.^ then 
there is absolutely no basis for any later determination as to 
whether the new Instructional system Is any better than the one It 
is designed to replace* This first step (1) is to "Identify the 
Behavioral Objectives" of instruction* They are called "behavioral 
objectives" because they must be stated In terms which are capable 
of objective measurement* Without the reliability of objective 
measurement there would be little purpose In going to the trouble 
of collecting data aimed eventually at making a decision as to 
whether there was in fact genuine progress toward building an 
improved Instructional system* 

Of course* most professors believe that what they teach* and 
what Is important* Is not subject to this kind of objective 
assessment* This Issue* and the skill involved In identifying 
behavioral objectives. Is the topic of the section prepared by 
Dr* C* F* Pauison* Dr* Paulson has been conducting research on 
methods for teaching the writing of behavioral objectives and has 
been actively developing Instructional systems for six years* 

He is thorou^ly familiar with the Issues. He Is considered as 
one of the national leaders In this rapidly developing Innovation* 
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Figure 1. Major Stages in System Design for Developing Validated 
Instructional Systems 
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The prevalent approach to stating objectives might be classl*- 
fled as an Infotmatlon approach, l.e. objectives are stated in terms 
that boradly outline a content area. The learner might be expected 
to "know" Information cn topic from Aachen (a German city) to 
zymase (yeast) and the professor would, through some maxiner, deter- 
mine the adequacy of this information about the topic and, even- 
tually, rate the learner® a performance. The difficulty with this 
approach is that for all practical purposes there is an infinite 
amount of Information that can be supplied on any object (and the 
quantity of this iaiformaticn is increasing hourly) . A more 
rational and effective approach must be developed to determine what 
is to be learned or ba faced with the prospect of 

having to teach more than can be learned in a lifetime— op every 
single topic. This Is^ossr' <le situation can be resolved through 
the Identlficatlop of behavioral objectives, and at the sas^ time 
it affords the basis for significant Improvements in learning 
effectiveness. 

Once the ultimate objective has been identified there is a 
requirement then to determine the sub-behaviors (or elements) 
Involved in that ultimate behavior. The analysis of any complex 
behavior reveals that it is composed of less complex behavioral 
elements which are essential to the effective and consistent per- 
formance of the more complex behavior. This second step (2) is 
shown in Figure 1 as "Determine Enabling Objectives." For 
exaoq)le, the task of driving a car (as an ultimate objective) 
involves knowing how to start it, how to stop it, how to steer it, 
rules of the road and many other "elements." Obviously, each one 
of these sub-tasks or elements can become a training objective in 
and of Itself. Therefore, it too is an objective, but It is 
merely an objective in the sense of enabling the more ultimate 
behavior to be performed. 

Figure 1 illustrates that from the analysis of the ultimate 
objective (Step 2), two other kinds of activities are generated. 

One of these activities (Step 3) indicates that identification of 
enabling objectives permits the construction of performance 
measures. While this is perfectly true, for the time being we 
think it may be useful to set this activity aside and follow down 
the vertical line of blocks to learn the other kinds of activities 
generated from determining enabling objectives. Step four (4) 
Indicates that when one has determined the enabling objectives it 
is possible to classify these into various types of learning. 

One type of learning might involve simple association, such as 
calling an object by a name, while a much more complex type of 
learning tolght involve decision-making or problem-solving where 
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objects oust be classified, evaluated, and manipulated in very com- 
plex ways. 

Depending upon the kind of learning that in Involved, it is 
possible to identify the kinds of events that provide the 
optimum conditions for learning each kind of behavior (Step 5). 
Once the events have been Identified that will permit optimal 
learning, then it is possible to further identify the form of 
those Instructional events and specify them in a manner detailed 
enough to permit the actual, construction of an instructional 
system* This process is somevhat analogous to that followed in 
the building industry* Step 1 may be likened to the baeic 
decision to build a house, a garage, or an office building* Once 
the ultimate purpose of the building has been established, it 
then becomes possible to analyze more specifically what is 
desired* Of course., analogy can be perfectly coiqparable but 
tbe general task of deterimining the vario\is kinds of spaces 
desired and their approximate sizes or proportions is similar to 
the task of determini^^ the enabling objectives. It too is an 
identification of the major elements which will comprise the 
total structure. Carxiying the building analogy through the next 
three steps (4, 5, sad 6), It cm be seen that the bedrooms, for 
exas^Ie, will have different kinds of requirements for the 
different kinds of people are to house* When these require- 
ments have been clarified, it will be possible to specify with 
considerable detail their size and location. And finally, with 
all these decisions having been made, it will be possible to 
construct a detailed blueprint which would specify in a very 
detailed manner exactly the form that each room must take in the 
finished product. 

This four-step process (2, 4, 5, 6) has been given the n me 
of “objective analysis and Instructional specification.” This 
topic hs8 been developed by Dr* Paia Twelkcr in Section III. 

Dr. Twelker is an educational psychologist who has been the 
recipient several research grants In the area of programed 
instruction* He has developed instructional systems and is par- 
ticularly adept: in analysis and planning learning experiences 
desig^c^d to produce specified types of behavioral outcomes* 

There are several conq>lex issues involved in these opera- 
tions. Many professors believe it Is impossible to efficiently 
pre-plan the kind of experience that will be maximally effective 
with a given class on a given day. Some feel it is essential to 
“develop the subject as it moves along.” There Is some wry gcod 
evidence on this issue and there is much to be said for the 



flexible point of vlev, but it is not in opposition to detailed 
analysis and planning* Being flexible is most effective as ft 
condition that comes after detailed analysis and planning— >lf 
optimal learning is to result* In other words » the analyses and 
the procedures taught by Dr* Twelker are useful and necessary in 
the development of new instructional systems y regardless of how 
the instructor might modify or supplement that system when it is 
finally used under optimal circumstances* 

From the "blueprint'' or the detailed "instructional specifi- 
cations »" it is possible to produce an instructional product or 
prototype for each enabling objective (Step 7) * This version is 
then tried out, or tested, with students (Step 8) to determine 
whether, in fact, the intended learner can exhibit phe desired 
behavior* Whether the learner can or can no^ perform the behavior 
identified in the enabling objectives, has, SQms very coiq>lex 
relationships to the behavioral objectives ofigl^lly identified. 

In some instances the analysis (Step 9) will reveal that the 
instructional specifications were in error* Under other circum- 
stances, they will reveal that the Instructional product is 
deficient* Under still other circumstances, the analysis will 
reveal that both specifications and the product are proper, or 
that th^ are both deficient* Depending upon the results of this 
analysis, the product is appropriately modified (Step 10) and 
recycled (Step 11) through the production and testing of 
additional materials, procedures, or systems until a product is 
designed that does enable learners to perform the desired 
behavior in an effective and consistent manner* 

Steps 7 through 11 have been given the title of "Instructional 
Systems Development*" Dr. Dale Hamreus develops this topic in 
considerable detail in Section IV* Dr* Hamreus has had extensive 
experience in devel^ ^lag instructional systems , and over a four- 
year period directea a major grant from the Ford Foundation 
involving approximately fifty instructional systems ranging from 
the elementary school through the college level in most of the 
academic disciplines— including the physical and social sciences, 
humanities, mathematics and even more generalized and inter- 
disciplinary subjects such as "use of the library" and "study 
skills." The Insights which he has gained over these years of 
experience hopefully will be useful in developing instructional 
systems that are valid, l,e*, teach the behavior they were 
designed to teach* 

Although the construction of performance measures was alluded 
to earlier, and is indicated as Step 3, discussion has been pur- 
posefully delayed to proceed through the steps in the developmental 




process In the most direct «aimera Figure 1. clearly indicates, 
however, that the performance measures are constructed very 
early in the developmental process and the irwo-vay arrows between 
Stages 2 and 3 indicate that the construction of performance 
measures actually aids in determining enabling objectives. This 

— 4bM j»aa^ mw juw4» wO wv uicaoM^sgUf WMV&is 

Is an analysis of the ultimate behavioral objec^ves. Further, by 
constructing performance measures early, the specific measures 
used to determine the quality of the ultimate behavior exhibited by 
the learner are identified, and these specifics are avoided in the 
instructional experience. Thus, an individual does not learn the 
answer to specific questions, or the solution to a particular 
kind of problem, in a rote way, but rather is required to solve 
a problem, or arrive at a generalization, or perform any other 
task in a manner which assures that he can perform the desired 
behavior in a meaningful context. Figure 1 also Indicates that 
the measurement devices are useful in the tryout of the product, 
and the analysis of results aids in refining the measurement 
instrument itself. 



When all steps have been completed, it can readily be seen 
that the instructional system is complete Including all necessary 
performance measures. Even the term "performance measures" is 
intended to comnunlcate that this does not necessarily refer to 
paper-and-pencil tests, but may Include any conceivable way to 
bring an objective assessment to the behavior being taught. 

The construction of performance measures is covered in 
Section V under the title of "Measurement." This section was pre- 
pared by Dr. Del Schalock. Dr. Schalock has spent the last five 
years developing new kinds of measures of complex behavior. Most 
of his professional career has been dedicated to the search for 
improved measures of the complex forms of human interaction often 
stated in our objectives, but somehow never quite measured in 
existing paper-and-pencil tests. The f list-tests" which he has 
designed constitute one of the newest and most promising approaches 
to one of the most difficult problems in measurement. 

At this stage, of course, the process of development has 
been completed. If one had complete mastery of the concepts and 
the skills described above, it would be possible to build a cost- 
plate instructional system that would be valid and would achieve 
the ultimate behavioral objectives originally identified. 
Unfortunately, the process is not that simple. When the time 
comes to adopt any Instructional system, the varying character- 
istlcs of learners, varying Instructional situations, teachers 



1-9 



that en^loy materials in varying ways, the varying context in which 
materials are used, etc. , all contribute to differences in the 
effects that a system may have. In order to test the system in 
these varying contexts, and to determine its effectiveness when 
employed in different ways, it is necessary to establish a 
’^research design** 

Inferences . 
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Dr. Jack Gordon explains the principal considerations for 
research design in Section VI. Dr. Gordon received his ?h.D. 
in Besearch Design and Analysis at Michigan State University. 

His professional work has concentrated on problems related to 
research methodology, and he is nationally recognized and con- 
sulted as an expert in this work. He has sought to prepare 
materials which will help you critically analyze proposed 
research design and thus eliminate the sources of invalidity in 
the conditions of an experiment. Hopefully, this training will 
enable you to critically analyze your own designs and to perfect 
them in order that you will be enabled to make valid inferences 
about the effectiveness of instructional systems which you pro- 
duce in various contexts of iise. 

Sections VII and VIII deal with analysis of data gathered 
when the research design is applied. Many professors whose basic 
discipline is not in the mathematical area hesitate to become 
Involved in these analyses because they believe it is beyond 
their competence. Dr. James Bealrd, who has prepared Sections VII 
and VIII, has such a command of these techniques that he believes 
the basic concepts and basic decisions as to which kind of 
analysis is appropriate for given kinds of questioris can be 
readily mastered by anyone who is willing to learn a few basic 
considerations. There is general consensus In this age of the 
computer that it is not essential to learn the actual skills of 
calculation. Such training for many professors may constitute 
a waste of much human effort. However, the basic skill necessary 
to specify the kinds of information desired are essential for the 
person who wants to design and conduct research on instruction. 

Dr. Bealrd has devised a unique approach to assisting the novice 
in data analysis to enable him to give the kinds of direction 
necessary to appropriate experts in order that even a novice may 
attain the kind of information required to draw the valid 
Inferences alluded to above. 

Finally, it is necessary to possess skills to obtain support 
fox both developmental and research work and to report on the 
effectiveness of such systems once they have been developed. 

These activities are described in Section IX under the topic of 
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"Proposal Preparation." This section has been prepared by 0r* Jade 
Cramford. Dr. Jack Crawford is a clinical and experlsental psy- 
chologist who has bad ertensive experience in educational and 
psychological research and has received nuoerous research grants. 
His extensive experience In both conducting and administering 
researdi has provided him some insights in "grantsnaiishlp" which 
may' be invaluable to you when you seek support for your work. 
Obviously, there are no pat answers, or formulae, to present, an 
idea in such a way that you will be guaranteed support. However, 
there are some rules which must be followed and there are some 
practices to avoid. These suggestions or guidelines can help 
in making a presentation that is favorably reviewed. 

During the next two weeks you will be deluged with massive 
amounts of information relevant to research and development. 
Perhaps one of the most useful things you can do to integrate 
this information is to try to organize it around some research 
idea or project of your own* It is suggested that you construct 
a brief outline for a research proposal (see Section It) and as 
each topic is developed apply it to your own project. Check it 
daily with the staff member leading your small group discussion 
session, 9nd when it has his endorsement, check it with 
Dr. Edllng for any additional comment to get a second reaction. 

A proposal so developed is really one of the most ttseful outcomes 
of the institute. If you have developed a sound research proposal 
by the end of the institute you will have demonstrated attain- 
ment of the ultimate objective of the institute— and one of the 
significant enabling objectives of the COBD program. 

The detailed official dally schedule of sessions follows: 

Good Luck 
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APPENDIX A 



Official Iiiatltuttt Schedule 



Sunday 
August 20 


Registration and dormitory check-in. 






8:00-11:30 a.m. 


1:00-4:30 o.ii.e 


7:00-10:00 n.m. 
bociaX Event 


Mfttylgy 
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Independent Stuay- 
Specify's Ob^tivaa 


Auguot 21 


(Ot. Edltat) 




Section Il-Prctsaaaion 
Evaluation 
(Dr. Crawford) 


Orientation 
(Dr. Edllng) 


Social Event 


Tuesday 
August 22 


Section X-Fractice 
Session-Specifying 
Objectives. 

(Dr. Paulson) 


Small Grp. Sesslon- 
Speclfy*g Objectives 
(Dr. EdXlag-Staff) 


Xnderpendent Study- 
ObjectlvvV Analysis 




Section Xl-Xndepend. 
Study-Spec. Obj’s. 


Practice Session- 
Specify'g Objectives 
(Dr. Paulson) 


Independent Study- 
Objective Analysis 


Wednesday 
August 23 


Section X-Practlce 
Sess.-Obj. Analysis 
(Dr. Twelker) 


Snail Grp. Session- 
Objective Analysis 
(Dr. Paulson-StafiE) 


Independent Study- 
Icstruc. Syst. Dev. 




Section XX-Small GpSt. 
Specif y*g Objectives 
(Dr. Paulson-Staff) 


Practice Session- 
Objective Analysis 
(Dr. Twelker) 


Independent Study- 
Instruc. Syst. Dev. 


Thursday 
August 24 


Section X-Practice 
Sess • -Xnstr . Syst . Dev . 
(Dr. Hamreus) 


Sm^Xl Grp. Session- 
Instruc. Syst. Dev. 
(Dr. Twelker-3tf») 


Independent Study- 
Measurement 




Section IX-Sisall Grp. 
Sess.-Obj* Analysis 
(Dr. Twelker-Staff) 


Practice Session- 
Xnstruc. Syst. Dev. 
(Dr. Hamreus) 


Independent Study- 
Measurement 


Friday 
August 25 


Section I-Practlce 
Sees . -Measurement 
(Dr. Schalock) 


Small Grp. Session- 
Measurement 

(Dr. Hamreus-Stf) 


Independent Study- 
Research Design 




Section ll-Snall Grp. 
Sese . -Xnstr . Syst . Dev . 
(Dr. Hamreus-Staff) 


Practice Session- 
Measurement 
(Dr. Schalock) 


Independent Study- 
Research Design 


♦At 4:30 p.m* to 5:00 p.m. daily there will be a general 
in the auditorium for both Sections I and II to evaluate 
raised during the day. 


discussion session 
and clarify Issues 



Saturday 
August 26 



Sunday 
August 27 









Official Institute Schedule 
(Second Week) 





8:00-11:30 a.m. 


1:00-4:30 o.m.* 


7:00-10:00 n.m. 


Monday 
August 28 


Section X-Practlce 
Seas .-Res. Design 

(Dr. ^rdon) 


&iall Grp . Session- 
Research Design 

/riy> . 


Independent Study- 
Data Analysis I 




Section II->SmJi.6rp. 
Sess . -Heasurei&ent 
(Dr.Schalock-Stib) 


Practice Sessioi- 
Research Design 
(Dr. Gordon) 


Independent Study- 
Data Analysis I 


Tuesday 
August 29 


Section I-?ractice 
Sess •-‘Data Anal. I 
(Dr. Beaird) 


Snail Grp. Session- 
Data Analysis 1 
(Dr .Gordon-Stf • ) 


In^epmdettt Study- 
Da^a Analysis II 




Section IX-Sml.Grp. 
Ses8.*Res. Design 
(Dr.6ordon‘-St£.) 


Practice Session- 
Data Analysis I 
(Dr. Bealrd) 


Independent Study- 
Data /vnalyala IX 


Wednesday 
August 30 


Section I-Practice 
Sess.-Data Anal. 11 
(Dr. Bealrd) 


Small Grp. Session- 
Data Analysis II 
(Dr .Gordon-Stf.) 


Ind. Study-Proposal 
& Report writing 




Section lI-Sml.Grp. 
Sesa.-Data Anal. I 
(Dr*6ordon-Stf •) 


Practice Session- 
Data Analysis 11 
(Dr. Bealrd) 


Ind. Study-Proposal 
& Report Writing 


Thursday 
August 31 


Section I-Fractice 
Sess .-Proposal & 
Report Siting 
(Dr. Crawford) 


Sml *Gxp. Sess .-Prop. 
& Report Writing 
(Dr. Bealrd-Stf.) 


Social Event 




Section lI-Sml.Grp. 
Sess.-Data Anal.ll 
(Dr.Bealrd-Staff) 


Prac • Sess • -Proposal 
& Report Writing 
(Dr. Crawford) 


Social Evei^t 


Friday 

Sept- 1 


Section 1-Flnal 
Criterion MeaSr 
(Dr. Crawford) 


Summary & Evaluation 
of Institute-Dormitory 
Check-out 
(Dr. Edllng) 






Section II-Sml.Grp. 
Sess .-Proposal & 
Report Writing 


Summary & Evalxiatlon 
of Institute-Dormitory 
Check-out 





(Dr.Bealrd-Stf .) (Dr. Edllng) 



*At 4:30 p.ia* to 5:00 p.m. dally there will be a general dlsousalon session 
In the auditorium for loth Sections I and II to evaluate and clarify Issues 
raised during the day. 

participants may make tra*/el departure schedules 
after 3|:00 p.m. Friday* Transportation to Portland 
and Salem terminals will be provided. 
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II - OBJECTIVES 



me 









I 






Evaluation Sheet 



We need to know how well the ideas and issues in this ssanual ate 
comsunicated to you* You ate the test audience for this material* 
To remove or strengthen the weak spots » to retain or Improve the 
strong ones (if any) , an account of your learning experlence^ja^ 
you read these secvtions is crucial. Use this sheet as you study 
and jot down your reactions* Don’t be concerned with typographical 
errors* We’re concerned with how the message is coming through. 

SECTION # TITLE . - 



Give topic 9 
paragraph or 
sentence, page 



Importance 

Rate 

1 2 3 4 5 
Low - High 



Clarity 

Rate 

1 2 3 4 5 
Low -High 



Suggested 

Improvements 



Over-all' ^Jitlng of the Section: Use a 5-point Scale 





Important 






Understandable 




1 

Very 

Low 


2 3 

(Circle one) 


4 5 

Very 
High 


1 

Very 

Low 


2 3 4 

(Circle one) 


5 

Very 

High 



PLEASE PUT ANY ADDITIONAL COMMENTS ON BACK 




SECTION II 

Specl£]riiig Behavioral Objectives 
Casper F« Paulson 



The Case Against "Specifying Obiectlves " 

An interesting phenooenon occurs whenever educators prepare 
to docunent plans for instruction* Inoiedlately prior to under** 
taking the business at hand^ they prepare eloquent stateinents about 
valueB they hold deatu and believe they share with their patrons* 
^piis activity is variously described as "specifying objectives 9*' 
stating goals or "describing aims*" The regularity of the 
occurrence of this phenoi&enon is difficult to understand f because 
only infrequently do these statements appear to have any relevance 
to the subsequent planning activity^ In most cases, the relation- 
ship a^Lpears to be analyagous to the playing of the national aptb ^. ^ 
before an athletic contest* 

purpose here to describe effective techniques for 
planning and developing instructional systems* It is aot our 
purpose to perpetuate dogma* Thus it has seemed necessary to 
examine critically even sudi a generally accepted procedure as 
specifying objectives*" Careful examination of the effects an d 
side-effects of the procedures generally subsumed taider this label 
indicate that their positive contribution is usually negligible. 
Worse yet, they frequently involve behavior that is psychologically 
maladaptive, socially repugnant, and directly antithetical to the 
improvement of instruction* 

Generally speaking* when someone expresses an intention to do 
something, to b^ike a cake, or build a home, or vacation in Hawaii, 
we take him seriously* We expect that he really will do what he 
says he intends to do* However, if a person continually expresses 
intentions to do certain things, but never does them, we soon learn 
to discount such talk as idle chatter* We come to believe that 
such a person has lost touch with reality, and that he achieves 
satisfaction from expressing dreams rather than achieving goals* 
Whether such behavior Involves merely fantasy or autistic thinking, 
in other words whether he is merely dreaming and knows it, or he 
really believes that he can change his circumstances by merely 
wishing hard enough, we tend to feel sorry for such a person an d 
regard him as a victim of himself, but harmless to others* 






Thor# Tflay be serious consequences, however, when statements of 
intention are taken at their face value and subsequently fxove 
Inva^^ld. If we hire a contractor to build a house, and later find 
^t he lades the competencies and resources to complete the job, 
the fact that he had good Intentions does little to lessen our dls- 
satlsfactloa. It Is even possible that the expressed Intentions 
wera nothing less than fraudulent# Expressions of Intent are like 
promissory notes. They are only as good as the ability and dater- 
miaatto of their maker to fulfUl them. The Implication that 

objectives range from daydreams to autistic 
accidental, nor Is the implication that taking them 
seriously may have harmful consequences. 

^fortuMtely, the actual achievements of instruction are much 
neasure than those of other enterprises. The 
fu ^ compounded when objectives are expressed In terms 
that make them Invulnerable to validation, as when they refer to 
e^nte to occur In the distant future or In the inner recesses of 
the human mind cr soul. We have been told that we will be able to 
separate true from false prophets by their fruits. It Is difficult 

^ credible that seeds of 

instruction will lie dormant for an extended period oi time, and 

tten ^ro^ously sprout, msture, flower, end bear fruit at some 

the distant future. It seems reasonable that true prophets 
attent-.on to the fruits by which they could be 

Imagine, If you will, the effect on the progress of medical 
acl^ce if we could not differentiate surgeons from quacks, phar*- • 
mclsts fr^ medicine men, antibiotics from sugar water. Imagine 
how much of our economic progress would have been possible If con- 
tractual agreements were entered Into with such expanslveness, and 
hmored with such prodigality, as is 8o commonly the case with 
educational objectives. If you can Imagine these things, you can 
have some general idea of how much the statements of educational 
objectives have contributed to the progress of education to date. 

1 of educational objectives are fundamentally srsud- 

ulent. They have the capacity to deceive both those erpressiag 
ttem and those to whom they are expressed. They may be used both 
to camouflage our confusion and to propagandize the public. 

to summarize the case against specifying Instructional 
objectives, not only Is this acavlty frequently an unproductive, 
even Irrele^t, ritual; It frequently Impairs Intellectual effec- 
tiveness and Impedes educational progress. 
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The Case, for Specifying Instructional Oblectives . 

Although the iioproper use of statentents of instructional ob- 
jectives has probably led to widespread skepticism and suspicions 
on the one hands and mlsplaqed confidence on the others the hard 
fact r4w*ns that any substantial and durable improveoient in in— 
struct^. ould probably be impossible without theqi« Just the use 
of the buim "improvement*' Implies that we will achieve sos^thlng 
more fully s frequently s or efficiently s or something not previously 
achieved* When we 4ave described this "something" and set out to 
achieve it* we have an "objective," 

The only way we can demonstrate improvement is to show that we 
have achieved at least one of the above. Such ixaprovemeats are 
quite imlikely to occur by chance. They will usually be the result 
of a concerted effort, Furthermores if improvements are to survives 
Md lead to further improvements s they must not rely merely upon one 
person's opinion^ but documented evidence that will persuade even 
the unconvinced. Mere opinions even convictions will not suffice. 
Learning achievements must be described in terms that are credible 
and significant to the users. 

Improvements of instruction is synonymous with improvement in 
achieving Instructional objectives. We can hardly improve our 
proficiency at achieving those objectives without improving our 
pfoficiency in defining them. While recognizing tlie multitude of 
pitfalls and abuses that are possible in stating objectives s the 
appropriate question Is not whether 9 but how* we should state them. 



What is an Obiective? 

At the very least* ar objective must represent some point or 
event that Is identifiable. You must be able to tell when you have 
arrived. If others f *e expected to pursue the same objective* they 
mist Uke«?i8e be able to determine when they have arrived, The 
number of potential objectives is* of course* infinite. They only 
become objectives when we set out to achieve them. But before we 
leave this point its Importance should be emphasized. Just being 
al!>le to recognize something of value can be of considerable impor- 
tance. 

We often read of cases where valuable stamps or coins or 
priceless antiques have been sold for ridiculously low sums be— 
(iause their owner did not appreciate their true value, Ea<di of us* 
during our lifetime* has probably had pass through our possession a 




numiber of coins tha^; could have been sold for considerably more than 
tbelr face value. IiL' Is quite possible that some person reading 
this has In his pocket one such coin* Yet only the trained ob* 
server I the Informed niimismatlsti will be able to capitalize on such 
serendipitous occurrences* It is unlikely that a person could make 
a respectable living b^^ passively waiting for such rare coins to 
pass through his posseiislon* The professional soon learns to 
Identify and pursue lll^ely sources* 

Thus we come to the next essential characteristic of an 
objective* There are an Infinite number of potential objectives* 
They only become real objectives when someone sets out to reach 
them* This Implies not only a committment to attain them« but some 
control over the conditions necessary for their attainment* One 
may wish for rain^ but unless he can cause it to rain» rain is not 
a legitimate objective* 

A true obJeetlve» then^ should have these characteristics: 

1* It is defined clearly enough that we can re<- 
cognlze It* 

2* We can carry out whatever activities are 
necessary to make It occur* 

3* We seriously Intend to achieve lt» even at 
considerable cost* 

Many of the loalfunctions of the objectives described pre- 
viously can be attributed to the fact that they do not fulfill the 
above criteria* Thus, many objectives are either not identifiable » 
or we lack the means of achieving them^ or we are not really com- 
mitted to their adilevement* Only when these three criteria are 
satisfied can we truly say that we have an objective* 



t^at Is a Specification? 

When we build a house there are some decisions that we are 
willing to leave to the architect or builder » and some that we 
hff/e made ourselves* When we supply a list of conditions « those 
things that must be Included^ the conditions that must be met» we 
have supplied a list of specifications* Many other conditions may 
be varied « but these conditions must be met* We must give certain 
specifications before the work can commence > but it Is almost as 









Important for us to realize what we should and should not specify. 

For eiouople. if we tell an architect that we desire that the 
home be comfortable and attractive, we have not yet made specifi- 
cations. He doesn't know exactly whht we mean by these terms, and 
he will request some more specific guidelines. Our specifications 
must be explicit enough to allow him to draw up a detailed plan. 

It is Important, however, that ve specify the right things. If we 
are too specific about the materials to be used and the manner of 
construction, we may impose conditions that are too restrictive. 

The most useful specifications that we could supply to an 
architect would probably be those that described how the house 
should function, rather than how it should be built. For example, 
we may wish to have entertainment zones sufficiently separated from 
the sleeping areas so that our children won't be disturbed When 
we have guests. We may have fondness for outdoor living and phobias 
against bathroom congestion. Supplied with such a list of 
functional objectives, and permitted sufficient latitude ‘In the 
means to achieve them, an architect may be able to provide us with 
a much more effective plan than we would be able to draw up for 
ourselves. 

Similarly, in constructing behavioral objectives it Is impor- 
tant that we be specific, but it is also important that we be 
specific about the right things. We should be much more concerned 
about the valued outcomes than about the means to attain them. 



Why "Behavioral” Obiectlves? 

We can specify objectives in several ways. We can specify 
those learning activities that we intend to take place, or the In— 
structlonal activities that we Intend to perform. But these are 
actually ozily means to an end, which is that the students will 
learn” something. The whole business of education Is concerned 
with changing people in some way, with developing characteristics 
that were not there previously. Unfortunately, the changes that 
we seek to accomplish are usually difficult to observe. Yet in- 
structional decisions, unless they are purely arbitrary, are based 
upon some kind of observations. Readily identified landmarks like 
topics covered, pages read, or time elapsed have a way of becoming 
pseudo-ob j ectlves . 

To avoid this pitfall, it is important that we render learning 
achievements more conspicuous. There is nothing terribly no^rel 
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about ttla approach. Progress In virtually any science depends 
Upon CQ6 to oxkhamco the obsetveblllty of phcs&osutoa of In** 

terest* The cloud cheiobep and the Geiger counter* lltisus naoer and 
microscopes, all serve this purpose. The real challenge ih con- 
structing behavioral objectives Is not to focus interest on what is 
Madi^ ^parent (as some skeptics think), but to make m^re apparent 
that iml^ la o* most Interest. This may involve structuring ime 
students environment so that their actions are more readily 
observed, or more carefully structuring out observations so that we 
are more sensitive to significant aspects of the students' behavior. 

It requires more than a little skill to construct objectives in 
such a way that they will provide the instructor with the iafor- 
DM^on that he needs to make Instructional decisions effectively, 
wiuiottt becoming Involved in such extensive testing that all of the 
intended learning is stifled^ A prudent gardener does not dig up 
his seed every morning to see if it is growing. But there is really 
not much difficulty in getting students to behave in an overt and 
obser^ble manner, as most teachers will testify. The task is more 
me of structuring the situation so that behaviors of Interest can 
be observed, and then determining whether such behaviors Indicate 
the attainment of the instructional objective. 



The Three Dimensions of an Instructional System 

There are three crucial factors that will determine the ultimate 
Impact that an instructional systems will have on the improvement 
of education. It suits our purpose hers to think of these as 
dimensions, and then to employ an analogy. All solid objects are 
considered to have three dimensions. Each of these dimensions is a 
factor determining the "size" of the object. If any of the dimen- 
sions is sero, the object sliLply doesn't exist. Similarly, an 
Instructional system can be thought of as having three dimensions, 
each deterging the "size" of its contribution to education. With 
the Tver's forbearance, we will employ the mnemonic device of 
identifying these three ''dimensions*' as "The Three D's of an In- 
structional Systems Design, Development, and Description." With 
reapect to each of these dimensions, statements of objectives are 
of crucial importance. 



Design. Whether we aspire to build 
instructional system, there comes a tine 



the house of our dreams or an 
to translate our aspirations 






into specific objectives* As anyone who has built a home can tell 
yon» this is the time ^en the dreaming stops and the hard decisions 

Ktfg4vi~ VliXX VSS^ 1 j[Vg 1 y ha S0 e3jS QU? 

aspirations until they are commensurate with our resources and, 
appropriate to our circumstances » this does not mean that we abandon 
cur dream in despair* On the contrary^ it is the time when we 
should guard most tenaciously those parts of the dream that we 
valued most and surrender only those valued least* How i^ell this 
can be accomplished will depend largely on the clarity and explii^ 
itness of our values* 

Similarly^ the planning of an instructional system should not 
signal the abandonment of philosophical goals » educational ideals • 
or esthetic values* In spite of all the indcind things tiiat have 
been said earlier about intangible goals and lofty aspirations » they 
are not inherently evil* When it comes f:o stating objectiveSf. thdy 
represent a very good place to starts but a very poor place to stop* 
The more dearly they are hcdd» the more urgent it becomes to express 
them in a form that will allow us to plan effectively for their 
Sthievement* Sach statements give us the general direction of the 
objective t but they do not describe m objective* It may be of In- 
terest to know that there is some excellent fishing east of heret 
but a rabid fisherman would want to know how to locate and Identify 
some specific excellent fishing hole* 

Just what design functions should a good objective serve? In 
addition to giving direction to the instructional task« it should 
delimit the task* It should describe something feasible of attain- 
ment with the resources at hand* A general goal such as appreci- 
ation of our cultural heritage not only reqidres transformation in- 
to more observable terms » but considerable delimitation if it is to 
be feasible of nttainmente No one would argue that describing the 
historical origins of the Bill of Ri^ts is all that is involved in 
an appreciation of our cultural heritagCt but It does represent a 
legitimate objective that is highly relevant to the more general 
value statement* 

Delimiting the objective allows the developer of an instruc- 
tional system to get closure* One could never complete the task of 
developing an appreciation for our cultural heritage » but the point 
of attainment of a behavioral objective is readily identifiable* 

If the behavioral objective is to serve the purpose of providing 
closure and identifying completion of the instructional task, it 
must describe behaviors that will be observable within the spatial 
and temporal limitations of the instructionc Furthermore^ its 
attainm^t should not depend, upon external^ uncontrollable events* 
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In addition to giving direction » focus | and reasonable limits 
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stated in a manner that facilitates the planning of appropriate in- 
structional procedures to achieve it* We must have at our disposal 
the means of attaining an objective^ or else it is no more a 
wish* In this events our Instructional activity 9 like a rain-dance^ 
functions to relieve our anxieties » rather than to achieve our 
objectives* 



This is one reason why it is important to express the objective 
in behavioral terms* There is no scientific body of knowledge that 
speaks to the problem of developing hypothetical constructs* But 
educators and psychologists do know something about cha ngin g human 
behavior* It seems to be fashionable to deplore the dearth of 
scientific principles of instruction* but the difficulty perhaps 
lies not so much in our deficiency of knowledge as in our defi- 
ciency in stating instructional problems appropriately* It may be 
true that this yoimg science is not yet possessed of ten talents* 
or even five, but this does not justify burying the talent that we 
do have* 

Concurrent with the planning of the operations of an instruc- 
tional system is the development of a test to determine whether the 
system achieves the stated behavioral objectives* This test differs 
from those used in conventional instruction in an important ways 
It is tssed to evaluate the instruction* not the student* Thus 
while the administration of a final examination usually signals the 
fulfillment of an instructor's responsibility to his students* the 
end of his task* the administration of the test usually signals not 
the end of the developmental task* but merely the inception of a 
new and crucial phase* 



Peveloament . 

”Xhe best-laid schemes o* mice an* men 
gang aft a-gley." 

The first trial of an instructional system will usually provide 
one with an opportunity to quote Robert Bums. It is rare Indeed 
that even the most carefully designed instructional systems will 
function perfectly on the first trial* Even though well established 
learning principles may have been applied meticulously in the desi^ 
stage* the isipleiaentation and cond>lnation of these often have un- 
expected or unwanted results* In a sense* the first draft of an in- 
structional system Is a kind of complex hypothesis to be tested* 









The ai^proprlate test of the system Is not whether It was presented 

eiM a* ^ A. — -• — « « « 
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The function of behavioral objectives In this context Is to 
provide a sensitive lndlcato<r of the performance of the system^ and 
to Inform the developer of the nature of its strengths and weak- 
nesses* Good objectives enable the establishment of an efficient 
Information feedback system that allows timely correction of the 
deficiencies of the system* It should be noted that the use of eim- 
pirlcal procedures in developing an instructional system places a 
uniquely valuable resource person » the naive student, on the devel- 
opmental team* Frequently Instructional planners know too much, and 
have forgotten how difficult It Is to learn* The student, by his 
learning performance, can shoi? the Instructor where too much pres— 
vious knowledge had been assuxied, where the Instructional pace is 
too rapid or too slow, and where the desired learning Is or is not 
taking place* 

It is the system, not the student, that it being tested. It is 
fairly well known that a good student can learn in spite of poor 
instruction* What is not quite so generally appreciated la that 
effective Instruction can overcome many learner deficiencies* 



Deacrlntion . 

**If my net don’t catch it, it ain’t fish*” 

This homely saying is often directed at behavioral scientists* 
suggesting that they are preoccupied with overt behavior and to- 
tally unconcerned with anything not presently amenable to obser- 
vation* No doubt many wondrous things slip through our nets, but 
that should not deter us either from fishing or developing better 
nets* But let us rephrase the saying t 

*'If my net hasn't caught It, I shouldn't try to sell it.” 

When the development of the Instructional system has been 
completed and its effectiveness demonstrated, it becomes important 
to describe it in terms of its most relevant characteristics so 
that It may used by others* If this is not done, the system 
constitutes an ix^tructlonal ”grab-bag»** 

The description should be Intellectually honest* This re- 
quires, in jfehe first place, that the system be described in terms 
of the instructional objectives actually attained and obse.cved in a 



a realistic trial# It is not appropriate or honest to describe 
’'intended" outcomes as attributes of the systen# T^Xe intentions 
nay be attributes o£ the designer | only the observed achlevenent of 
these intentions is appropriately regarded as an attribute of the 
system# 

Intellectual honesty is further served when we waVa clear our 
educational goals with explicit statements or operational definitions 
of just idiat we consider to be adecjuate evidence of the achievement 
of those goals# It is not sufficient to say that our students have 
developed an understanding of the democratic processes ^ even If our 
observations have convinced us that this is true# While most people 
would agree that such an objective is importantt there nay be wide 
diaagreenent about what behavior constitutes achievement of the 
objective# One Instructor may consider that he has adiieved the 
objective when his students have memorised the Constitution# Others 
may disagree idioleheartedly# The most honest and least deceptive 
approach is to describe the instructional system In terms of the 
observed attainment of behavioral objectives# If a potential user 
considers these objectives and this terminal behavior to be in- 
appropriate# well and good# At least he has not been mislead by a 
decepti^ label# It is up to him to determine# in his own case# 
the relevance of the Instructional objectives to his instructional 
value system# No one else is in a position to make this decision 
for him# 

No matter how effective an Instructional system is# its ultimate 
impact on education will be determined by how generally# and how 
appropriately# it is used# Ideally# both Instructional tools and 
instructional problems would be described in such a manner that they 
could be matched one with another appropriately# Unfortunately# 
this Is not presently the case# Tools of instruction# whether 
books# filmstrips# or motion pictures# are usually described and 
evaluated In terms of their physical properties# Catalogues list 
the number of frames in a filmstrip# the length of a mot'^on picture 
and whether it is black and vlilte or color# and so for:.^ Yet the 
ultimate purpose of these tools is that they work some in the 

students isho use them# It would seem that a far more appropriate 
criterion for selection and a far more useful mean? of description 
would be tlie behavioral rather than the physical properties of these 
tools# The prudent car buyer will assess the performance charac- 
teristics of a car by taking it out on a trial drive# He will 
observe how it handles#, the smoothness of the ride^ and the quiet- 
ness of the motor# The eeloctlon of Instructional tools ou^rits no 
less caution# 



The ABCD's of Writing Behavioral Objectives 

We have alacuSBed the function of behavioral objectives at 
length to underscore the crucial role that they can play in the 
development of an Instructional system* Nov It is time to 
consider the form that these statements should take* As far 
as putting objectives Into their appropriate form la concerned » 
the task can be described as simply vriting sentences wherein 
each of the elements^ subject, verb, and modifiers, has 
certain desired characteristics* We shall again use a mnemonic 
device, as the above heading Indicates, to ensure that the 
Information presented here. If not memorable, will be at least 
recallable* 

Audience* The "subject" of the sentence used to describe 
a behavioral objective should describe who is to be doing the 
lea rnin g* The description should be in terms most relevant 
to the instructional task at hand* Sometimes this description 
will be simply an identification of the grade level and the 
subject of a class, for eriample, seventh grade geograpl^ 
students* 

It is necessary to exercise care here, however, because 
the developer of the system is already making some commitments* 
Does the system really propose to achieve the behavioral object- 
Ives with all seventh grade geography students? Experience 
indicates that there are Indeed few objectives that will be 
achieved by all students* For one reason or another, some 
students inevitably fall by the wayside* If the objective is 
to be attained, by all, the students, then the system will have 
to provide for students with low reading ability, lov7 intelli- 
gence, or poor background in the subject matter* If the system 
is designed for students with average or better reading ability 
or intell^ence, and certain required subject maiter skills, 
then the audience" should be so defined in the subject of the 
sentence* 

While it is Important to provide sufficient relevant infor- 
mation in describing the audience, it is almost as important to 
avoid being compulsive about it* Unnececioary details should be 
avoided* Learner populations can be described in great detail, 
if you put your mind to it, but it takes time to write such 
descriptions, and time to read them* The writer should only 
provide such descriptions as will be useful. 



11-11 



Behavior, It may seem self-evident at this point that the 
verb In a sentence that descrlhea a behavioral objective is 
perhaps the most crucial element. A short cut us^ by those 
who edit and evaluate behavioral objectives is to skim through 
them 9 looking at the verbs. The use of inappropriate verbs is 
the most frequent » and most disabling » error in writing behav— 
loral objectives. 

The verb should describe an observable action that the stu* 
dentswill demonstrate as a result of their learning experience. 
Some verbs denote readily observable actions » others do not. It 
Is difficult to visualize Just what a student looks like when he 
is understanding;, appreciating, or even listening. But it is 
fairly easy to observe whether he is or is not speaking, writing 
or constructing. 

The following list of behaviors may prove to be helpful, 
Desmond Wedberg has asserted that all beh<ivloral outcomes can he 
classified in one of these ten categories. Whether or not this 
is the case, the list is at least suggestive of useful verbs. 

1* Identify 

2 « 

3. Order 

4. Demonstrate 

5. Describe 

6. Construct 

7. State a rule 

8. Apply a rule 

9. Interpret 

10. Distinguish 

CondltlosB. It was indicated earlier that the purpose of 
behavioral objectives was not so much to focus our Interest on 
what is observable, as to render more obseirvable what is of 
interest. The degree to which we svacceed in this will be deter- 
mined largely by the care and Ingenuity employed in describing 
the conditions of performance. It is here that we describe the 
setting for the evaluation, the materials and aids the student 
will be given, and the nature of the problems with which he wiU 
be confronted. It is liaportant to note that this section of the 
objective does not deal with learning conditions, but with 
evaluation conditions. 



If the behavior called for in the objective Is to identify 
each state of the United States, then an appropriate phrase 

n»aa«^ wwa«%M>w*W4M9 | w a» V wm u wAoiiaw uw»^ 

of the 48 contiguous states, on which only state boundaries are 
indicated., •••" But sometiines considerably more Ingenuity will be 
required in devising evaluation situationcu If the Instructional 
objective is something as s^^btle as Increasing interest in read- 
ing poetry, it may be necessary to describe a free reading sltua* 
tlon, where the teacher systematically but unobtrusively notes 
how many of the students In the class choose to read poetry* 

Other conditions may be appropriate for the same objective. A 
situation could he described where poetry hooks along \^th others 
were made available to the students and a log was kept of the 
books checked out. 



Uhile the most fri^quent source of failure of an objective is 
inadequate description of beiitavior, the difference between an 
ordinar]f and an excellent objective is frequently the amount of 
ingenuity exercised in describing the conditions of performance. 

Design . It seldom happens that the behaviors described in 
an objective are completely mastered by all students. To be sure, 
there are occasions when perfect mastery is required, and instruc- 
tion will be continued until it is attained. But most often we 
are willing to tolerate a certain variability of performance and 
are satisfied with some reasonably high, but less than perfect, 
degree of attainment. This portion of the statement of objectives 
establishes the decision point where the instruction is deemed to 
have fulfilled its purpose, and a new objective can be pursued. 

If the degree of attainment is set too high, it may require 
an extravagant use of instructional resources. It may even render 
the developtent of such an instructional system economiccdly 
unfeasible. This is a matter that can be settled only by the 
careful weighing of a nuzober of values. The experience of the 
instructor as to what constitutes a realistic expectation of 
students, and the extent to which mastery of this objective is 
prerequisite to subseqxieiat attainment of other objectives, must 
be considered. When the developer of an instructional system 
prepares this portion of his objective, he should recall that It 
is he, not his students, who is coomiltted to achieve it. Failure 
cannot be resolved by assigning grades. It requires a reworking 
of the Instructional system. 



Source Materials for Behavioral Objectives 

The behavioral objectives for an instructional system do not 
emerge by a process of spontaneous generation, nor are they 
entirely the product of creative imagination. Fortunately, there 
are a number of sources aad procedures that the writer of behav* 
ioral objectives can employ, that will simplify his task and 
contribute considerably to the outcome. A few of these will be 
mentioned here. The resourceful writer of objectives will 
probably discover some others. 

It is usually advisable to gather information from several 
different sources, rather than depending too heavily upon one. 

For example, it will frequently be observed that a professor will 
verbalizv^ one set of objectives, teach to a second, and test a 
third. One can only guess which set of objectives is the most 
valid. Choosing onl^f one of these sources may well result in an 
unacceptable set of objectives. 

The most useful, sources ox information will depend upon 
whether the instructional system is Intended to replace some 
conventional Instruction already being given, or is Intended to 
develop new understandings and skills not r resent ly taught; 
whether the objectives are primarily intellectual or manipulative; 
and whether the writer is working on his own instructional prob- 
lem, or serving as a specialist for an Instructional problem 
where the subject matter is outside of his area of competence. 
Following are some suggested procedures that should be helpful 
in most sltuatiovis. 

Verbalized Obi ect Ives . Perhaps the first stop in preparing 
a set of behavioral objectives is to luterview the person or 
persons who understand best what the outcomes of the instructional 
system should be, or who have the greatest stake in its success. 
Even if one is writing objectives for an Instructional system 
that he will build for himself, it is advisable to begin by expres- 
sing the objectives as articulately as possible. 

A frequent mistake is to "behavloralize*' the objectives too 
soon. This is particularly true when the writer is interviewing 
a subject matter specialist who is not accustomed to thinking in 
behavioral terms. Forcing the interviewee to speak in your terms, 
rather than trying to understand him on his terms,, can easily lead 
to fnist ration and loss of cooperation. 
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An efifectlve teclmlque 8e£3&s to be to gxradually "zero la” on 
behavioral descript ions. Only after becoming fsmillar with the 
instructor's value orientations, his philosophical goals, and 
instructional alms, should one try to become more specific. The 
early part of the interview can supply numy valuable clues as to 
\diat behavioral objectives vlU be conslclered most; relevant. If 
the Instructor is forced to supply beliavlortU descriptions too 
soon, he may merely naixov his attention to those outcomes most 
easily described, and never reveal those outcomes that are of most 
Interest to him. 

Once you have detertalned the "direction" in ^ihich the objec- 
tives lie, it is time to move in. for a more explicit description. 
One can ask, "What does success look like?" "What does a student 
do that indicates to you that he has really learned what you 
wanted him to learn?" "If the student could do this and nothing 
more, would you be satisfied?" "What else could he do that would 
he just as good?" "How reliably should a student be able to do 
this? Say, how many times out of ten?" "Should every student be 
able to do this?" "What proportion of the students?" 

The interviewer should have little difficulty generating 
meaningful questions of this sort, that don't confuse the inter- 
V’^ewee with unfamiliar terminology. The answers will frequently 
come easily, but sometimes only with great difficulty* It will 
be helpful to explain why the answers are important* 

It is usually neither feasible nor advisable to attempt to 
arrive at final statements of behavioral objectives in the inter- 
view, certainly not in the first interview. These can be written 
independently, and later offered for approval. Lengthy discussion 
about specific wording problems may arouse defensiveness, lead to 
futile digressions, and obscure more important information. 

One should not expect too much from the interview, or the 
interviewee. Productive interviews are taxing at best, and much 
of the information required can and should be obtained from other 
sources. 

Instructional Materials and Activities . Course syllabi a nd 
assigned reading materials can provide a considerable amount of 
raw material for writing objectives. They specify the elements 
to be learned, but usually reveal little about the way the student 
ij intended to behav'^. when he has learned them. They do, however, 
give the writer a comprehensive view of the range of objectives 
that will be required for the system. 




11-15 



tor to ‘•y ooservlng the activity of an Inattuc- 

detewtoe his appraisal of the relative value of different 

?* natter, and to datemine v*at hind of iMmioa 

r^vior ne etpects from his students. He taay. for 

tM^M “°“f” aataUed factual data, but .aphasite L^yei^ 

?“ ‘J® explicit, if not the most tellable, tiidlci" 

^t L outcomes that the instructor considers ilimor- 

taat is the procedure he uses for evaluation. ^ 

Evaluations. Since test performances do represent 
i^5w”®v 5“*^®°“®* would seem that they are a natural place to 

InrolvS **“®*® la « certain amount of risk 

^ ’ because the nature of moat tests la Inflaenced 
hesvily by factors other than instructional obi actives Thexr 

^sure the most testable rather than the moat significant outcomes 

^ construction, 

doing Ite Quite nacurally, considerably more energy 1b devoted to 
the preparation of instruction that, to the prep^ILn oH^M. 

"‘® instructor that a teat be eaw trcoM^t 

® distribution of scores ’ 

^ srss.“,2S4"jrs,r,t:.ss. 

evaliiatifm^ ^ acceptable performance. The conditions of 

lnS^ed^« observed may suggest conditions that should be 

included In the statements of behavioral objectives. 

be develonf^I*^?* the instructional system to 

desirable^^ achieving psycho-motor skills. It Is 

wv ^ ^ ^ ® number of Individuals performing the task 

both su^essfuUy and unsuccessfully. It Is hard to analyze a com- 

behavioral components If only successful 
performances are observed. But If one can observe a varietrof 
«nau:ces»f«l perform«.cea as ueU, the varlou^lIL^^fJrthar 















. ® 

ERIC 



being detissned uiifiatiiB^iactpty suggest the variety of behavioral coa* 
pomicta required for successful pierformance* By thus breaking down 
one complex behavloi’al objective Into a number of simpler behavioral 
objectives 9 the function of the objectives in facilitating planning 
and development of an Instructional system is enhanced considerably* 



Frenarlng the Statement of Objectives* 



The procedures described above should yield more than enou^i 
Information for writing the objectives* 'fhe task now becomes oxie of 
organising and simplifying* Discrepancies and Inconsistencies will 
have to be resolved » wd behavioral statements examined to sea If 
aaytlilng has been lost in the translation* Irrelevant data must be 
eliminated, and assumptions questioned* Even when this has been 
accomplished, the task of actually listing the objectives may seem 
formidable, particularly for more extensive Instructional systems* 



While the statement of Instructional objectives must be both 
compr^enelve and explicit. It does not follow that there must be 
a separate statemen*' for each objective, nor is It necessary to re- 
peat compulsively elements that a number of objectives have in 
common* For example, if all of the objectives are appropriate for 
a given population of students, the audience need be described only 
once In a prefatory statement* Objectives to be measured under 
similar ccmditlons can be grouped together, and the conditions des- 
cribed but once* In some Instances, It will not be feasible to 
measure all intended behaviors for all students* For example, if 
oae hundred definitions are to be learned, a sample of these 
defirdtlons can be tested and the proportion of that sample that 
constitutes an acceptable degree of achievement can be specified* 



Introduction 



"We learn to do neither by thinking nor by doing, but by 
thinking about what we are doing*" 



George D* Stoddard 



This paper has urged that gpals have primacy over procedures, 
and that the latter be evaluated in terms of their contribution to 
the former* It would indeed be an Irony If the procedures sugges- 
ted here would themselves be accepted uncritically and employed 
unquestlonlngly • 
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EjufectlvenesGi In writing behavioral objectives will be achieved 
f.xQm exwlaed e^erieuCe* what has gone before is merely a preface • 
meaningful only if it leads to such experience* Thus it seems more 
appropriate at this point to introducet rather than conclude or 
suomarire* 

The procedures and rationale presented here are wortliy of 
tee ting I but not blind acceptance. They are no more inviolable ^n 
the practice of beginning a paper with the Introduction* 
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Evaluation Sheet 



We need to know how well the Ideas and Issues in this manual are 
consamlcated to you. You are the test audience for this material. 
To remove or strengthen the weak spots » to retain or improve the 
strong ones (If any), an account of your learning experience 

4 a TTa^ o oe mvatt 
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and jot down your reactions. Don't be concerned with typographical 
errors. We^re concerned with how the message is coming through. 

SECTION # TITLE 



Give topic , Importance 

paragraph or Rate 

sentence, page 1 2 3 4 5 

Low - High 



Clarl.ty Suggested 

Rate Improvements 

1 2 3 4 5 
Low -High 



Over-all Rating of the Section: Use a 5-point Scale 



Important 

1 2 3 4 5 

Very Very 

Low High 

(Circle one) 



Understandable 
12 3 4 

Very 
Low 

(Circle one) 



5 

Very 

High 



PLEASE PUT ANY ADDITIONAL COMMENTS ON BACK 
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SECTION III 

Objective Asalyais esd Instructional Speeitie^tlon 

Paul A* Ti^elker 



Overview 

t 

In a systems approach to instruction^ the systems designer 
(or "instructional engineer") has the task of creating and 
sequencing a series of learning experiences which will produce a 
previously stated behavior# To achieve this end* the first thing 
the systems designer must do Is to define the Instructional ob- 
jectives or desired terminal performance outcomes# You are by now 
familiar with the processes Involved In stating behavioral ob- 
jectives# Nhen a systems designer states what he wishes the 
student to learn In behavioral terms | he not only provides a means 
for looking at the ins true tl<mal content in a facllltative manner » 
but he also provides a means for determining the criterion 
measures required to test x«hether or not the students have learned# 
By specifying the instructional content in behavioral tetteSf the 
designer provides himself with Information that pertains to the 
design of Instructional methods and materials# This Information 
is obt^ned through processes whldi we shall call "objective 
analysis" and "Instructional specification#" 

The Instructional methods and materials » when Implemented 
according to some administrative instructional plani become the 
Instructional program— after eidiaustlve try-out and revision# At 
the completion of instruction# criterion measures are applied to 
obtain an Indication of the adequacy of the program# An Integral 
step In the development of a new instructional system Is the 
provision of self-corrective feedback all along the way that is 
used to modify the system If the desired effects have not been 
achieved# The learning experience Is continually monitored and 
empirically developed until effective# 

In summary# the process Involved in designing an instructional 
system may be analyzed into four steps: (1) determining terminal 

objectives and sub«*objectlves (or enabling objectives) through 
objective analysis (l*7hat shall I teach?); (2) determining the In- 
structional conditions (How shall I teach It?); (3) developing 
the Instructional conditions (Nhat*s wrong with my prototype# and 
how can I correct It?) ; and (4) measuring the results of In- 
struction (Did 1 teach lt?)« The topic of objective ana3»ysla 
falls within the first step# while the topic of instructional 
specification falls within the second stap# 



Predetlgn Extemporanao'us Design 

^ j ^ j« ^ ^ <a — <4 
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issues involvei in the design of instfuctlonaX nsterlals in the 
manner described above* Frankly | ve want you to be aware of these 
issues and at least accept our point of view long enough to under- 
stand it« In some future time» you nay xdLsh to try it ovit* 

Instructional ccRiditicus may be preuesi^ued or th^r may be 

designed **on the spot” while instmction proceeds« The latter 
method is employed by most teachers as ”just tell:hig or showing 
the student what you* want him to know*” "Extemporaneous design of 
the conditions of Isaming* during interaction with a student or 
studentst is undoubtedly considered by many teachers one of their 
most Important functions ( 7 $ pp* 251)* . Programed instructlonel 
materials • on the other hand^ Illustrate intentional predesign 
of the conditions of Instruction* 

There are argum^ts both for and against the predesign of 
instructional conditions* An obv/lous argument against predesign 
is that it is too time consuming for the average instructor* 

Loaded down with a full schedule of courses > an instructor simply 
does not have the tine to develop an instructional system* An 
argument in favor of predesign, is that the pre-selection of 
proper instructional conditions nay be made in an unhurried nanner 
rather than "on the spot*” Think about the natter of predesign 
vs extemporaneous design# IJhat are the advantages of predesign 
over extemporaneous design? What are the disadvantages? Consider 
the feasibility of each method# Jot your ideas down below* 

Dlsadvcntagos of Predesign of Instructional Conditions 
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Advantages of Predesign o£ Instructiooal Conditions 






Other Factors to be Considered 



GsgadT cites several Important advantages of predesign of In- 
structional conditions « They include the followings 

**1. The selection of proper learning conditions may be made 
a» an unhurried choice, rather than in *spur of the 
moment'' decisions* 

2* A 'quality control* of the choice of learning conditions 
is ensured and maintained* Quality does not suffer from 
variations in teachers' skills* 

3* Predesign makes possible pretesting* Whether or not a 
set of learning conditions has been correctly chosen and 
designed can be determined by trying It out on students, 
and revising if necessary* 

4* Predesign of learning conditions greatly reduces the 
necessity for the teacher to use valuable time in 
extemporaneous design, and thus makes It possible for a 
proper emphasis to be restored to the teacher functions 
of managing instruction > motivating* generalizing, and 
assessing” (7, p* 253)* 

Try to find the time to read Gagnd's discussion of predesign 
(7, pp* 250-254)* It will give you greater insight Into the 
advantages of predesign* Gagnd claims that it appears to be 
completely feasible and well within **be limits of an instructors* 
time restrictions* This is a point that sometimes provides a 
catalyst for fierce arguments In favor of extemporaneous design* 

It should be pointed out, however, that knowing what to teach 
(knowing the objectives of Instruction and all the related sub- 
objectives) bjlngs the Instructor a long way toward specifying in- 
structional conditions and eventually provides the opportunity for 
empirical testing and "validating” the effectiveness of the in- 
structional conditions* 

In your list of disadvantages of predesign, you may have 
mentioned that extemporaneous design better takes Into account the 
individual differences of students* You might have argued that 
with extemporaneous design, the teacher can detect differences of 
individual learning and react to them extemporaneously during In- • 
structlon* Gagnd contends that this Is an unconvincing argument* 
He doubts that the typical classroom provides a place where the 
task of designing Instructional conditions for Individual students 
can even be accomplished* Because of high load requirements, the 
teacher simply can not deal with each student individually* 



Further^ extemporaneous design night produce ins true tloaal con- 
dltione that are too slow for the better learner » might move at 
the proper pace for the average learner • and it might miss the 
slower learner coa 4 »letely« Gagnd’ also points out that predesigned 
instruction may take individual differences into account* You may 
wish to pursue some of tiiese points later in the small group 
discussion period* 

Relation Between Research and the "Instructional Systems Approadi" 

It should be noted that our present state of'4tnowledge 
exhibits more of the characteristics of an art than a technology* 
However^ we can not go so far as to say that there are no 
principles or procedures that might guide our efforts* Rnfortun- 
atelyt might find that some of these principles may be more 
readily applied to specifying the instructional conditions for 
science courses than for humanities courses* This is simply to 
say that we cannot be dogmatic when it comes to developing in- 
structional conditiOi&8* The procedures noted below are valuable 
only as they help the instructor reach his goal* 

The guidelines presented below do not give the instructor 
hard and fast rules for specifying the series of learning ex- 
periences whl(di will produce a previously stated behavior* What 
the guidelines are designed for is to help the instructor look at 
the problem of instructional specification in a systematic and* 
hopefully, benaft^l. mannsr* All of these guidelines may not be 
new to you* Ceftalnlyt a competent instructor utilises many of 
the procedures listed below as he attempts to prescribe the learn- 
ing experiences for a course* Further « these guidelines do not 
et all inhibit an instructor's creativity in specifying instruction* 
Rather t It Is hoped that the guidelines will enable an instructor 
to be optimally creative* In other words » what is presented 
below are not answers for the Instructor who is grappling with the 
problem of instructional specification— rather » they are 
suggestive of appropriate questions an instructor should ask as he 
alts down to determine what and how he should teadi* 

What relation does the specification of instructional 
conditions have to your development of research hypothesso? How 
can the procedures mentioned below concerning instructional 
spaclficatlon help you in deciding idiat to research in terms of 
your own instruction* To paraphrase Gustafson (10 » p* 10) t 
neither objective analysis* instructional specification* or even 
the determination of training objectives can significantly advance 
the state of the art in education* Genuine improvements can 
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evolve only throttrii gaoirlcaX reaearch conducted In accordance 
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vAfch aotmd experimental practices ■ However» the Instructional 
systems approach can help clarify problems and bring the experience 
and ingenuity of instructional specialists to bear more directly 
on then* The proc^'*^ ^ ^redesigning Instructiony Which must be 

repeated anew fo\^ f '< t rue and each research projecti has the 

duel effect of (1) i..> th 2 application of subjectivity to 

only those areas uheva ?;he state of the art makes subjectivity 
necessary • and <2) clarifying the issue so that the state of the 
art can be brou^t into play most effectively* 

171th the aid of an instructional specialist and a madia 
specialist I the Instructor brings to hear on the question of in- 
st.'^ictlonal specifications all that he knows g and all that Is 
known by others (within personal limitations) about the subject* 
Only ^en may he conduct research In a meaningful manner* The 
nunmer in which research hypotheses may be derived Is tauoht to 
every jiraduate student* Confronted with a problem* he lists all 
ot the alternative nypotheses he can think of* and than tries to 
systematically disprove each hypothesis* either by citing theory* 
empirical evidence* or common sense* Those alternative hypotheses 
that are not discarded in is manner are examined more closely 
as probable hypotheses to research* This procedure* in effect* 
pro^dea a powerful tool for tkie instructor who Is grappling with 
the problem of Instructional specification* With the aid of other 
experts* he attempts to discard as beat he can all of the alter- 
native hypotheses ha has specified concerning an instructional 
problem he faces in his courses* The p«.'**adures of stating 
objectives In behavioral terms deriving the sub-objectives* and 
determining the instructional conditions* aid the instructor in 
sensitizing himself to those areas of instruction which require 
his greatest attention* 

With this procedure In mind* examine the following guidelines* 
and note how the procedure of setting up alternative hypotheses 
and systematically attempting to discard them fits within each 
guid e l ine * The following section deals In the prerequisites that 
must be considered by an Instructor before he designs an ' 
instructional system* 

Prerequisites to Oblectlve Analvsls.^.and Instructional Specification 

A prerequisite to objective analysis and instructional 
opacification Involves the Identification of ^’constraints” on the 
instructional system (these are prescribed limitations on what and 
how you can teach)* For example* some obvious constraints arei 
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(1) couwe descriptions, (2) ntabet of students, end <3) 
flnanctttx Take a few mcanents tp think about other c^tralnte 

that you as an instructor must consider as you specify 

Hons foic 006 of your courses* List as many 

of these constraints as you can* 
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Some constraints that you listed might fall 
under these headingss (1) information sources^ (2) 
resource personnel available » (3) d^ftlpsant and 
materials available ^ C4) time limits ^ (5) character** 
Istlcs of students, (6) facilities, and <7) trade- 
offs such as cost-effectiveness, time*-effectivenes8 
and so forth* 



After an Instructor has a clear idea of the nature of his 
students and the nature of the constraints placed on the in- 
structional system, he is ready to determine ^^t he is going to 
teach. Thus is the topic of the next unit. 

Objective Analysis 



The statement of instructional objectives is but a first 
step In determining Instructional requirements and has been dis- 
cussed elseidiere* Not only vould it be desirable to check the 
initial statement of Instructional objectives, it is also 
Important to determine what the student specifically needs to 
learn. It is unrealistic to assume that the flrat attempt at the 
sta;;ement of behavioral objectives will result in a list that will 
be kept unchanged for any length of time. Something as important 
as objectives cannot be treated superficially. They are not 
Witten so we can file them away unchanged once they have been 
specified. In relation to the second point— determining what the 
student should learn— there most be. some means to bridge' the' gap 
between where the student is at the beginning of Instruction and 
where he should be upon completion of Instructlcn. That is, 
we must specify ''enabling objectives"— objectives that consist of 
the component actions, knowledges, skills, and so forth that 

the student to attain the terminal objectives specified. 
Enabling objectives may consist of basic factual and conceptual 
knowledge serving as background information necessary to attain 
the terminal objectives. The tool that we may use to specify 
the enabling objectives and the necessary background information 
nwy be termed ‘'objective analysis," This analysis of 
subordinate uehavior is based upon procedures used by Gagne 

whl<di is sometimes termed hierarchical analysis or learning set 
analysis (6, 7, 9). 



For some subject xoatters, the enabling objectives or sub- 
ordinate kniiwledge may be arranged in a pyramid type of structure 




as shown in Figure !• In *uch a structure, layers of eonpeteaciee 
are identified* The basic prscsdurc is simple* Starttog with the 
teminal objectives stated in behavioral terns, the followius 
question is asked: ”What kind of capability would an Individual 
have ro oossess if he were able to perform this objective success** 
fully, were we to give him only instructions?*® The phrase re- 
gardlng instructions needs soipe clarlficetfop* .The learner,, ohould 
bet (1) told the form of the answer (e*g* nuaerlcal, sysbolic, or 
enacted) ; (2) Informed of any definitions that would clarify 
meaning of the objective; (3) provided with guidance st^gestlng 
the application of previously acquired Information to the. ob- 
jective under consideration* 



Enabling Objectives » - -■J 

Figure 1» A hypothetical hierarchy of 
enabling objectives for a single terminal objective* 

In the case of the hypotheticai. example in Figure 1, the 
answer to the questloxif **What would the Individual have to know? f 
it turns out, Identifiee a small list of major items of knowledge 
or skill* These are represented by A-> B, and C, on the figure* 
Tien the same q^stion Is asked for each of these items In turn, 
and so on until each item of knowledge and skill derived from the 
objective analysis has been added to the structure* Note that 
the enabling objectives thus gained are not the same perfonBjnr^r 
as the final task from which they were derived* They are In some 
sense simpler and in some sense more general* By using the 
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procedure of objective en«ly8l8| ve find out that ^at we are 
defining is a hierarchy or suborder of knowledges or skills » grow- 
ing ittcrsasingly Simla- In such a structurei options exist with- 
in layers as to seqi^ce of acquisition • but acquisition of erne 
layer in its entirity is called for before the presentation of tiie 
next hi^er or aore complex layer* For example | A-1 might be 
taught before or after A-2 is taught^ but each of these must be 
taught before A is taught* Some courses involve instruction where 
the structure of content Is relatively flat« While other courses 
represent hierarchies of skills as shown here* However » in both 
hierarchical and flat atrccture there may be many options possible 
in the sequencing of the objectives for the course* 

An example of a hierarchy of enabling objectives for a topic 
in elementary non-netric geometry is shown In Figure 2* The topic 
consists of knowledge which a student can use to specify setSf in- 
tecaeetions of set8» and separations of sets* Note that at the 
lot#est level (VI and V)| the concepts of separation of entities 
into groups^ point and set of points- must be known or acquired 
before the learner is ready to attain the hi^er order enabling 
objectives* lake a few minutes to study the hierarchy and prove 
to yourself that the lower order objectives must be attained prior 
to the higher order objectives* 

You may also wish to examine tiie question of what happens 
when learners actually undertake to acquire a set of principles 
that appear to have this hierarchical structure* Does the mastery 
of a Icfwer order objective actually affect the learning of the 
next ''higher" objective^ as would be expected? This question was 
tested in a study with sixth graders* The results of the study 
are summarized below* 

"The results showed that the learning of ^higher-level* 
principles was dependent on the taastery of pre- 
requisite 'lower-level* principles in a highly pre 
dictable fashion* For examplet of the 72 students who 
performed correctly on Principle Ila* only one did not 
perform Principle Ilia correctly on the test* Of the 
18 students who did Principle IXa incorrectly} all 18 
did Principle 111a incorrectly* The prediction that 
learning Ila depends on knowing Ilia was borne out} 
therefore} with a frequency of 99 per cent* For all 
the other possible comparisons} • * • the frequency 
of correspondence between predictions and findings ranged frop 
95 to 100 per cent* The learning of organized knowledge} 
according to these results} appears to be predictable 
fxon the pattern of prerequisite principles that make 
up the hierarchy of knowledge to be acquired" (7} p* 153)* 
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The ccmcXuslon cited above has been verified in a nuaber of 
studies using topics in matheoiatics (6| 8| 9)« 

Before goint on to another example* take another look at the 
hierarchy in Figure 2« Find the box ladled XIC| "Identify and 
dra:^ the intersections of lines or parts of lines taken two at a 
time I as nore than one point*" Belov^ list the enabling objectives 
that the student siust know hov to do in order to be inst^cted in 
this principle* Include only those enabling objectives that are 
pertinent for the student attaining lie* 
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Figure 2* (Caption on page 13)* 



Answer 



By asking the question! "What would the student 
kova die jLs Owdisf lie isi 

this principle?") you will find that he must know 
principles Ilia) Xllb) and IIIc* Further he must 
know IVa) IVb) Va» Via and VIb because XIC) the 
higher«order principle) incorporates these lower- 
order principles in the sense that the learner must 
be able to perform these operations before he can 
progress to the ne^ct level# 



Another hierarchy of knowledge for two tasks is ihowu in 
Figure 3* Take a few minutes to trace through the hierarchy* 
Identify all of the lower-order enabling objectives necessary for 
the student to know in order to satisfy lla and lib* List ^ese 
objectives below* 

lla m 

Enabling Objectives Enabling Objectives 



Figure 2* Hierarchy of enabling objectives 
for a topic In elementary non- 
metric geometry* Adapted from 
Gagn^ ( 7) p* 150 )• 
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Figure 3# Hierarchy of enabling objectives a topic In 
matheioatics* Adapted from Gagn^, ec al«» (8) 
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togwer 
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In order to accomplish XIa» the learner must be 
able to state and use the definition of addition of 
an integer and its additive inverse (Hia) as well as 
using the whole nwber 0 as the additive identity 
(IVe)* In order to accomplish IXb^ the learner must 
be able to accomplish all those behaviors stated in 
Levels TV. aad V« 

! 

Now take a look at Figure 4« Examine the terminal objec* 
tive and note that three components must be known before the 
student can perform the objective* The first (la) requires that 
the student be able to identify the forces acting on a body in 
opposition to each other* In order to learn this principle » the 
student must first know IXa^ that ls» the conditi<ms for' equili** 
brlum* 

Examine the two enabling objectives listed at the bottom of 
the figure* Then ask the question* "Nhat would the student have 
to know?" for objective Ic* Which objective fits in the frame 
labeled Xlb? 



Which objective is a prerequisite for Ilia? 



Check your answers by examining principle Xb* Does the enabling 
objective you picked as Ilia represent a prerequisite principle 
for Ib and not for Xc? 
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Figure 4* Hierarchy of enabling objeetlvea for a physics 
topic* (Adopted xrom Gagnd^ (7 9 pp* 155)* 
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Ansv/er: 



Objective 1 belongs in the frame labeled Ilia and 

Cbj active 2 belongs in Xlb« Being able to identify 

parts of a triangle is certainly a prerequisite to 

identifying trigoneaetric relationships in triangles » j 

I 1 

Below are listed five enabling objectives. If they look 
familiar » it is because they are taken from Figure 2 . But don't 
look back at that figure until you have tried ordering these 
objectives. Try to consciously ask the question^ "Wliat would 
the student have to know?" with each objective rather than simply 
ordering the objectives from the complex to the specific. Order 
these objectives beginning with the terminal objective. 

Specify the intersection Order 

set of a simple circle 

and parts of lines Terminal Objective Cl) 

Identify and draw a set 
of points 

Identify and draw a simple — 

closed Cjurve 

Identify and draw a .■ 

point 

Identify and draw a curve .. 

The correct order appears in Figure 2. Check your answer by 
tracing the hierarchy for Objective Id. 

It is a relatively simple task to order enabling objectives » 
asking the question, "What would the learner have to know?" It 
' is a more diffia:lt task to derive the enabling objectives from 
your own objective analysis. Rarely will you be fortunate enough 
to have the enabling object.' /es already formulated and diagramed 
for your particular terminal objective. Take a few minutes now 
to attempt an objective analysis on a behavioral objective of 
your own. These will be discussed during the small group 
discussion period. If you have any questions as a result of this 
exer^rise, jot them down. Bring Uiem up during the small group 
discussion session. 

Sometimes the question is asked, "If an hierarchical analysis 
is done correctly, can there be more than one structure obtained?" 
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To quote froa Gagne, "There are perhaps several possible learning 
set hierarchies vhich could be worked out • . • , and it is quite 
conceivable that some are •better* than others in a sense of being 
more efficient or more transferable to later learning" (9, p. 5)* 
In the long run, any hierarchy that is developed is "good" only 

^l«A 4>Uta4> 4*> . .« _« 

— ' — - «.M»w *«. ciAMg buc jkUDb&ucsoc aja aecelnoixiuig wxiat to 

teach, and in what sequence to teach it* 



Another way to describe the organization of enabling ob- 
jectives is by using an outllnb. The terminal objective dia- 
grammed In Figure 1 could be outlined as follows: 



I* Terminal Objective 

A. First level enabling objective 

!• Second level enabling objective 

2« Second level enabling objective 

B. First level enabling objective 

C« First level enabling objective 

!• Second level ena. iing objective 

a« Third level anabling objective 
b« Third leval enabling objective 
2. Second level enabling objective 

3« Second level enabling objective 

What are the drawbacks in representing all hierarchies in 
outline form? (Hint— see Figure 2 or 3). 
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Answers 



As shown by figure 2 and 3» some lower-order 
objectives **tle in** to more than one higher-order 
objectives. A hierarchy that is diagramed shows 
these relationships more easily. 



It is most useful in org.inizing objectives to type or print 
each component on a 3 x 5 inch card and sort these cards on a 
large table. ^You may find the chalkboard too inflexible to be of 
nrudh value.) Vhen the objectives have been arranged t they may be 
diagramed. 

VJhy go through the rather tedious procedure of objective 
analysis? One reason for arranging enabling objectives in a 
sequence of some sort at the outset la to consider the extent to 
vhich learning of one objective may facilitate the learning of 
another » or at least may have a bearing upon the design of in- 
struction to be specified. By laying out the structure of en- 
abling objectives in such a manner, the Instructor may readily 
determine if he has been teaching too little or too much. Teach- 
ing too little will cause the student to require further training 
while teaching too much might raise the cost of Instruction and 
require longer course time* After an objective analysis is com- 
pleted, it makes good sense to ask the question, **Can sti^ents 
perform the action in the terminal objective without this parti- 
cular skill or knowledge?** If the answer is In the affirmative, 
that skill or knowledge may not be an enabling objective, at least 
not for some students. Enabling objectives are derived from a 
knowledge of both the terminal objectives and the existing 
capabilities of the student population. In some courses, the in- 
structor cannot assume any abundance of prior knowledge at the 
start of instruction, while in other courses, (for example, gradu- 
ate courses) the instructor assxunes an abundance of prior knowledge. 
The limit to which an instructor details the structure of the ob- 
jective analysis depends upon the entry b^vibr of his students. 

A second reason for conducting an objective analysis Is that It 
serves as a self-corrective means to check the behavioral ob- 
jectives as initially stated. The Instructor has a means ot test- 
ing vdiether each objective is feasible to teach in a given amount 
of time and in the context of learning experiences designed for 
other objectives. For example, it is quite feasible that basic 
knowledge components are prerequisites for mcra than one objective. 
Knowing this, an Instructor may capitalize on this fact and teacdi 
these particular objectives in sequence. 




Vben aa ixistructor or researcher has taken the time to 
conduct an objective analysis, he is well on his way toward 
knowing what to teach, and in what ae(;^euce to teach it« Ob- 
jective analysis lo a technique that, when 
applied by the Instructor wi^ little or no help from others, 
save aO occasional conference with a coUeague to check on 
questions concerning the subject matter* When an objective aaaly- 
aia is completed, the Instructor essentially lias determined 

is in a position to specify how to teach it* 
Tliis is the topic of the next unit* 




DetexBinlng...The Instructional Conditions 

Identifying for Each Objective the Type of Learning Represent^ 
ed* It would seen that an important basis for matching instruc- 
tional objectives with instructional conditions would be to 
draw upon the large amount of Infomatlbn on human learning that 

liCU» waafc — — ^ 

classified into a category which is homogenous with respect to 
the conditions fostering learning of' that type, the Instructor 
would har-i at his command a very powerful .tool with which to 
specify instructional conditions* Eckstrand (5) *'.as suggested 
that such a taxonomy would provide ”a system for organizing 
existing Information in a way which would facilitate its appli- 
cation to particular training problems” as well as provide ”a 
most useful tool in determining deficiencies in our knowledge 
and thereby serve to guide future research*” Several attempts 
have been made at specifying this learning taxonomy (e*g*, 4, 7, 

12 « ) Although each of these .taxonomiea leave snichvtor 
be desired, Gagnd*s taxonomy seems to be the most promising* 

Gagne' < 7 ) has identified eight ^pes of learning* As 
these are identified below, keep in mind that the purpose of 
this taxonomy is to provide a system for specifying as best we 
can the different set of conditions necessary for the learning of 
each of the objectives previously identified* 

The eight types of learning that Gagne dlstinqulshes are 
as follows: 

Type 1: Signal Learning * A general, diffuse re* 

sponse is made to a signal (l*e«, the classical 
conditioned response of Pavlov)* 

Type 2: Stimulus-Response Learning* A precise 

response is made to a discriminated stimulus (i*e*, 
trlal-and-error learning, and instrumental 
learning) • 

Type 3: Chaining* A chain of two ot more 

stimulus-response connections are acquired by the 
learner (e*g*, procedural skills)* 

Type 4: Verbal Association (a subvariety of Chaining)* 



Verbal aasociatlon Involves the learning of 
chains that are verbal (e«g«« paired associate 
learning such as the naming of objects)* 

Type 5« IhiltiPle Discrlnlaation* Different 
identifying responses are made to as many dlfferest 
stimuUi i^ich may resemble each other in physical 
appearance to a greater or lesser degree* Xt is 
simply e .matter of establishing nuttbars of 
different chains ehen the connections tend to in* 
terfere with each other*s retention* 

Type 6; Concent leamlne^ eapeblllty of making 
a eomnott response to a class of stimuli that may 
differ from each other widely In physical appearance 
is exhibited in concept learning* 

7: Principle leamine* Principles are cheine 
of concepts that are generally termed knowledge* 
Principles represent the relationships among con- 
cepts* 

Type 8: Problem Solving* T^ie kind of learning 
that requires "thinking'* may be termed problem 
solving* Essentially* two or moi^ previously 
acquired principles are used to produce a new 
"higher-order" principle that results in a 
solution to a problem* 

It would be Impossible in the short time we have to discuss 
erch of the eight types of learning in detail* and to expect you 
to master each type sufficiently so as to recognize examples or to 
generate your own eiMples* In fact* instructors faced with the 
task at hand will usually secure the services of an instructional 
specialist who Is well acquainted with the various types of 
learning and the conditions necessary to bring the learning about* 
For purposes of this institute* however* it seems appropriate to 
at least look at the more important types so that you will have a 
better understanding of the processes involved in designing an 
instructional system* 

In college instruction* the most frequently occuring types of 
learning probably are concept learning* principle learning* and 
problem solving* This is not to say that the simpler types of 
learning do occur In college Instruction* For example* in foreign 
language classes* multiple discriminations may have to be 
established* Training certainly may occur in activities such as 
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pronouncing foreign words* 

Below are descriptions of the three most frequently occurring 
types of learning* They have been taken from Gagne and simply , 

present the highlights 6f the characteristics of each type of. 
leamlnge 

"Learning Type 6l Goacept Learning 

Attention now tuims to a kind of learning that 
appears to be critically dependent on internal 
neural processes of representation for its very 
existence* In man^ this fisnctlon Is served by 
langiiage* Although a number of animals have 
been shown to possess the capacity to make in- 
ternal representations of their environments i 
present evidence suggests that this capacity . 
is extremely lliolted even In the higher apes* 

Hunan beings f in contrast t employ this capacity 
freely and prodigally; they are highly inclined 
to Internalize their environment, to 'manipulate* 
it symbolically, to think about It In endless 
ways* 

Learning a concept means learning to respond 
to stimuli In terms of abstracted properties 
like ’color,* ’shape,* ’position,* ^number,* 
as opposed to concrete physical properties like 
specific wave lengths or particular intensities* 

A child may learn to call a two-inch cube a 
’block,' and also to apply this name to other 
objects that differ from it some^jdiat in uize 
and shape* Later, he learns the concept cube* 
and by so doing is able to Indentlfy a class 
of objects that differ from each other physically 
in infinite ways* A cube may be represented 
concretely by objects made cf wood, glass, wire, 
or almost any material; the object may be of 
any color or texture, and of any size* Con- 
sidering this great variety of physical 
stimulation that may correctly be identified 
as ’cubical,* perhaps it Is not surprising 
that it takes some very precise language on 
the part of geoi&eters to define what is meant 
by 'cube** But, of course,^ a person does not 
have to understand such a definition in order to 



III-23 



Identify correctly e cube under most ordlnat^ 
conditions of bis exl^itenpe* Except for some 
very special purposes of mathematical theory, 
an individual Identifies a cube ^Intuitively,^ 
that Is, on the basis of an internalised 
representation that does not employ the words 
of the geoiaeter's definition* whatever the 
process may be, there can be little doubt that 
a concept like cube Is learned, and that Its 
possession enables the Individual to classify 
objects of widely differing physical appear* 
ance* His behavior cbmes to he controlled, not 
by particular stimuli that can be identified 
in specific physical tenui, but by abstract 
properties of such stimuli* 



As an example, we may consider how a child 
learns the concept middle* Initially, he 
may have been presented idth a ^t of blocks 
arranged like this: O 13 0« If 

previous Ss^R learning has enabled him to 
receive reinfurcement for a request such as 
'Give mu a block,' he can then. readily learn 
the simple chain of picking up the middle 
blodc when his parent says 'Give me the middle 
one*' Similar chai^ns can then he established 
with other objects, such as balls arranged in 
the suae configuration Q O# sticks 
I II* One might want to make certain that 
the chains were generallzable over a range 
of separations, like } | jand j | | * In 

other words, the deliberate attempt Is made to 
establish a number of chains applicable to a 
variety of specific physical configurations* 
Continuing this effort, the spoken word 'middle* 
might be applied to various other arrangements 
of objects. Including such situations as theses 
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By this means • the child comes to respond 
correctly to middle as a concept meaning *nn 
object bet^feea two others** (Other meanings 
of *middXe,* such as the ’center of an area^' 
are* of course* different^ but may be similarly 
learned*) 

4 

Hov QCQB one know whethei the ehlXu hoS In 
fact learned the concept middle? The crucial 
test is whether he w-ii be able to respond 
correctly* not by chance* to some new con- 
figurations of objects he has not previously 
\ised in the course of learning* for example* 
these might be: 



If the child can now respond properly when told 
'Give me the middle one** it may be concluded 
that he has learned a concept* and that his 
b^avior is not controlled by specific stimuli*' 
(7* p. 47-49). 

"Learning Type 7 : Ptineij^le Latming 

As an approach is made to a description of ;he 
kind of learning that can be identified as most 
broadly applicable to the content of formal 
education* the job grows somewhat easier* The 
most difficult kinds of learning to describe 
are the simplest kinds* perhaps because it is 
necessary to he so careful to keep them simple* 

A kind of learning like principle learning, has 
some well-known conditions for Its establish- 
ment* which all of us employ very frequently* 
Continuation of the theme of the arg'^munkt— 

that different varieties of lea Ing can be 
distinguished by the conditions required to 
bring them about— leads to the view that 
principle learning is not ouly a highly familiar 
but also readily understandable kind of learning* 
As Is true with certain forms* the most important 
conditions are the prerequlsltles for learning* 
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Prlnclplie learning may be eoramplifled by the 
acqulsltlcm of the 'Idea*^ contained in auch 
propositions as 'gases expand when heated' ; 

'the pronoun e^f ^ takes a singualr verb* | *salt 
is composed of the elements Ka and d * ; xa 

definite article die goed with 
a feminine (Gerzean) noun' ; and many^ many others. 
Surely there can be little doubt that hisaan 
beings tsust learn large numbers of such principles^ 
some Interconnected » some not^ along the road to 
attaining the status of being considered 
educated adults. 

Possibly the immediate demurral will be raised. 
*Uhy. these are simply verbal facts to be 
memorized''^ Hot so$ and that is why the word 
^dea^ has been employed to describe these 
principles, inexact as that word may be« From 
our previous discussion, it is apparent that 
each of these statements can be l^med as a 
verbal chain* If we want to teach a five-year* 
old to memorize them in that way. this surely 
can be done. But the only kind of performance 
that would be possible following sudi learning 
would be something like this; Complete the 

statement: 'The pronoun each takes , 

. ' Such a performance is by no means what 
is established when one has learned a principle* 

In fact, one may learn the principle about 'each* 
and its verb without being able to verbalize 
such a statement at all. ^at by 

learning such a principle la this capability: 
being able to use the singular form of the 
verb in a variety of sentences or clauses having 
each as a subject. If an Individual is able 
to demonstrate this capability in a number of 
instances of representative sentences and clauses, 
one is justified in concluding that he has 
learned a principle* 

In a formal sense, a principle is ^ chain 
gyp or more concents. Some would say. It Is a 
relationship between concepts; but it seems 
preferable to state the nature of this rolatlon- 
ship* The simplest type of principle may usually 
be cast In the form 'If A. then as in the 
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tfcaaple 'If a femlnlnii nouni then the article 
p. 51-52) • 



"Learolns Type 8: Problem Solving 



Once he has acquired some principles f the human 
being can use them for many purposes in dealing 
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also do something else that Is most importanti 
he can think* Basically^ this means he Is able 
to coo^ino the principles he has already learned 
Into a great variety of novel higher-order 
principles* Be may do this by stimulating him- 
self* in parti and also by responding to var loins 
forms of stimulation from his environment* 



By means of the p'^cess of combining old 
principles into new ones* he solves problems. 
that are new to him* and thus acquires still a 
greater store of new capabilities* The 
problems he solves are neif to himi but they 
tsay not be new at all to other people* Of 
course* every so often a scientist or other 
creative person may arrive at a problem solution 
that la novel to society in general* 



Problem solvingi by which Is meant 'thinking 
out' a new principle that sorabir > previously 
learned principles | is a process that is very 
familiar to most adults* There is nothing 
very 'special* about such events i since they 
are likely to occur with a frequency of ten 
or twenty times dally in the life of an average 
man* !^en a driver maps his route through 
traffic (as opposed to simply being swept along 
by It) he la solving a problem* T^en a man 
replans his luncheon schedule as a result of a 
new appointment, he is solving a problem* TJhen 
a housewife decides to shop selectively for 
certain ^items on the basis of differential 
prices, she is sol«lng a problem* These every- 
day examples bear a close formal resemblance 
to the problems that are solved by students In 
composing reports and themes, in marshaling 
arguments to present a point of view. In per- 
forming laboratory experiments, and in reading 
a properly written textbook* 
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Suppose that a student ef physics encounters 
this situation In reading his te^ithooks 

Power is defined as the work done 
divided by the tisae during which 
the work is doM. That is^ 

Power «■ 

t 

Suppose we have a body doing work on 
another body^ acting with force jP 
and velocity v. Can you show that 
the power delivered to a body is the 
product of the force acting on it 
and its velocity? 

The solution of such a problem requires thinking^ 
which fisight go some^at as follows. First, 
the student formulates to himself the fact that 
he wants an expression relating power to force 
and velocity (v). He then recalls that 
work is related to force and the distance 
through which it moves, that is. 

Power - ^ 

' t t 

and he is therefore able to state that 
Power « Fv 

and to recognize this as the formulation he is 
seeking. Involved in these steps, of course, 
is the use of other principles he has pre<* 
viousXy learned, pertaining to^ the substitution 
of equivalent quantities in equations* 

When these steps have been accomplished, it is 
reasonable to stippose that the student lias 
learned something new* He has not learned 
simply that Power* "W* because he could readily 
have learned that as a verbal sequence (type 4)* 
Rather, he has learned how to demonstrate 
in concrete terms the relationship of power to 
force and veloci^, beginning with the definition 
that power is work done per unit time* In 
other words, he has learned how to use this 









definltlo&t and how to generalize It to some 
considerable degree to situations In which a 
force acts against a body with a known velocity. 
This act of problem solving f thenj has resulted 
In soTie very substantial learning* The change 
In the Individual's capability is Just as clear 
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of learning,” (7» 



4«* 4« 4m omo 
«»<» 

54-56). 



other variety 



In a short paregrapht state the main distinctions between 
concept learning, principle learning, and problem solving* 



Aaswert 



In summary I we may say that when a learner res- 
ponds to things or events as a class | what is in- 
volved is concept learning* ^o or more concepts 
may be linked together to make up principles* 
Principles represent the relationships among con- 
cepts or among other principles* ^en the learner 
uses principles to achieve some goal* or \dien a 
higher-order principles is generated from other 
principles* problem solving is involved a 



Below are listed several examples of each of the three types 
of learning discussed* Attempt to classify them as elthers 
(1) concept learning* (2) principle learning* (3) problem solving* 
Several of the examples are shown with the particular 'type of 
learning represented to help you cut* 

Ir, A child learns to call square objects made of metal* glass* 
wood* and other material* "cubes*” 

messs. teasBlas 

2* A student uses . the singular form of a verb in a variety 
of sentences or clauses adding each as a subject $ 

principle learning 

3« The student demonstrates that the power delivered to a 
body is the product of the force acting on it end its 
velocity* 

problem solving 

4* A student can select a ruler and count the number of times 
it can be laid end to end along a linear dimension to be 
measured* 



5» A child points out objects that are blue from many objects 
different colors* 

6« A child shows that he can use a ruler to detenoine 
whether two objects are the same length when he is not 
allowed to compare their lengths directly* 
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7* A learner explains the fact that the bones of a whale have 
been found in the Sahara Desert* 



8* A tea<dieA' points to an object that a student has not 
previously seen# The instructor asks* "t^here is the 
edge?” and the student points to it* This is an ex^ple 
of 



9. A student fonaulates an answer to the question, does 
hot air rise?” 



10* A student is presented with a nuoiber of objects, some of 
which are round and some of which are square* The in- 
structor asks, **Which of these will roll?” and the student 
responds by rolling the round object* 



11* Given instructions to do so, a student identifies and 
draws a point* His behavior shows that he kncirs the 

of "point*” 

12* A student answers, tl^e question, ”In how many ways can 
identical jobs be filled by selections from 12 different 
people?” 



13* A student lists each different ordering of a set of 
objects* He shows that he knows the - 

of "permutation*” 

14* A student is asked to distinguish between two lists, one 
of which includes permutations and the other, ocoiblnations* 
He does so correctly, and has shown that he knows the 

of "permutation*" 

15* A student is shown and identifies several examples of 
striated muscle* He is then shown a new example and ue 
responds, "This Is striated muscle*” He has shown 
h'S knows the "striated muscle* 

16* A student is asked to demonstrate and describe how 

striated muscles work* Using a model of nn arm and the 
muscle connections, he shows the movements made When the 
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muscld contracts* He shows that he Iolows tha . 
of how striated 'muscles move an arm* 

A learner is asked to mark the muscle connections and 
the direction of movement of limbs of a froy* This Is 
an escample of - - * 



17 * 





Anawert 



Check your mwtvs TdLth tihose listed belowt 



4* Problem solving 
5* Concept learning 



6« Principle learning 



7« Problem solving 
8« Cuncept learning 
9« Problem solving 



10« Principle learning 




Concept 

Problem solving 

Principle 

Concept 

WMUVO|P«» 

Principle 
Problem solving 



Identifying the Instructional Events that Provide General 
Conditions of Learning for Each Objective # After Identifying 
the type of learning represented by each enabling objective « the 
task Is to develop an Inatrxictlonsl environment whldi vill trans- 
form learners Into greduates who can perform at the specified 
levels after instruction Is termlnated« To accomplish thls^ in- 
structional sequences should be designed that reflect the con- 
ditions of learning previously identified for each objective* 

A summary of the conditions for principle learning as cited 
by Gagne are listed below* If you wish to st\idy the conditions 
of principle learning In greater detallf a discussion taken from 
Gagne Is presented In Appendix A« For discussions of conditions 
of learning for tiie other types included in 6agn^*8 taxonomy | re- 
fer to the book* The Conditions of Learning. (7)* 

*'?he requirements for instruction of principles whether 
practic^ by a teadierf a filmi or a text book (are): 

Step I : Inform the learner about the form of the per- 
formance to be expected when learning is completed* 

Step 2 : Question the learner In a way that requires the 
reinstatement (recall) of the previously learned concepts 
that make up the principle* 

Step 3: Use verbal atatements (cues) that will lead the 
learner to put the principle together^ as a chain of 
concepts • In the proper order* 

Step 4; By means of a questlopf ask the learner to 
"demonstrate" one of more concrete Instances of the 
irlnclple* 
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Step 5» (QptlaaaU but useftd foy. Utar. laftOa^g-tteAlt 

By a suitable question » require the learner to snake a 

verbal statement of the principle" <7| pp« 149) •. 

As a college instructor! you probably do not teach 
principles in isolation as described above* Xnsteadi you teach 
a nuodser of principles! often without ever stopping for a breath 
in between* Consider! for ejcample; the principle stated in Step 
1 above* Rarely would an instructor question the learner to make 
sure that he knew the concepts of "inform!" "learner!*' "fonat" 
"performance!*' and so forth* Indeed! the instructor usually does 
not even assume that the learner would not know these concepts* 

But this is precisely where misunderstandings occur in instruction* 
If a student does not understand even 'one concept Included in a 
principle! the probability of hie success In understanding that 
principle (demonstrating one or more concrete instances of the 
principle) is lower than if he knew that concept* 

The essaminatlon of the conditions necessary to bring about 
learning of principles been presented simply as an example 
of what an instructor! together with an instructional specialist! 
might go through In specifying instructional conditions* You 
are encouraged to study Gagne's book! The Conditions of Learning* 
for detailed discussioj!.s of the conditions of learning of each ' 
type discussed! as well as a book by Briggs (1) which gives con«* 
Crete Instances of the application, of the techniques discussed* 

Identifying the Form of the, Instriietlonal 5t»ent* At this 
point! decisions have to be made as to the nature of the in- 
structional events required* For exantpl'?.! should they be verbal 
or non«verbal! and should they bo visual or auditory* Of course! 
there are other considerations in special cases such as the 
requirement for motion, duration of exposure! and so forth* 

Below! for each Instructional event involved in principle 
learning! are listed some options as to the form of the in- 
structional event* The example is taken from Briggs <1| pp* 33)* 

Instructional Event 

1* Inform learner about per- llay require objects or 
formance required* pictures (depending upon 

objective) « 



2* Stismlatlng recall of 
component concoptsc 



Hay be done by oral or 
printed speech; actual 
objects or pictures may 
be desirable (depending 
upon objective)* 



3* Verbal cueing* 



Oral or printed words* 



4* Appraisal* 



Hay require objects or 
pictures (depending upon 
objective) « 



Let*s examine an example given by Brlgge (If PP* 60-61) la 
detail* Although the example involves kindergarten childrenf the 
steps shown are applicable at all levels* 



Ob-^eetive: Demonstrate a method for using a 

neaSuri^ stl^ to determine whether two obJehtP' 
separated by* spacsf are the same length* 



Type of Learning : Principle learnings *Two 
* t objects equal to the same thing are equal to 
each other*' 



Conditions of Xeamiagt 

(a) Individual concepts must have previously 
been learned ('loogerf' 'shorterf' 'same 
length')* 

(b) The principle must be stated to the child 
and demonstrated! or it must be discovered 
by the child* 

(c) Several examples of application should be 
experienced to be sure that more than the 
verbalization of the principle is learned* 



»SteP-l 



Step, 2 



HgP.,.1 



Medlt Proeram i 

^P»ty^ctional Event? 

X« Zilfoxm le«m«r 
about perfoYoasee 
required* 



2« Stimtilate recall of 
eonponents concepts* 



3* Verbal Prompting 



4* By discovery or 

prompting Is stated: 
If a movable doi/el 
is equal to two 
others* as sho\m by 
side-by-side place-' 
ment* the tt;o others 
are the same length* 



Medium 

Tt7o seta of dowels are 
mounted on separate wall 
displays* Each set has 
members of different 
lengths* Problem is how 
to identify the members 
of the sane lengtii in the 
two sets* (They cannot 
be moved*) 

' k 

Child is led to recall 
how to find out if lengths 
are equal by side-by-side 
placement of objects* 

*Here are some loooe 
dowels that can be moved* 
Would they help?^ 

Children use loose dowels 
to xnake side-by-side com- 
parisons to identify 
equal members in the tv70 
display sets of dowels* 



Por this objective* also* media alternatives may 
he selected ft:: either individual or group instruction* 
For group instruction* the two displays and tiie loose 
do^^els are viewed and matched* Verbal statements by 
the teacher present the performance sought* and either 
provide or prompt the statement of the principle* 
Children make several matchings » using different movable 
sets of dowels* straws pencils* (Again* a motion, 
picture might present this principle* but having real 
objects to manipulate might accelerate this application 
of side-by-side placements*)* 



Recall that up to this point* the instructor Is 
Ing in cooperation with an Instructional specialist t 



still fTork- 
Once the 



instructional program has been developed to this pointy the in<* 
structor can proceed on his o^m to develop the program in terms of 
the actual words used to convey information* Of course » he will 
probably use the services of a production specialist if slides « 
overhead transparencies « or movies are required* 

Here la enother example ahowiiig how the Inatryctiopel con** 
dltlons are specified » given the conditions of learning* This 
example has been taken from Briggs (I 9 pp« 94‘*95)« 

Principle: The sun rises in the east because the earth 

rotates toward the east* 

Media Program : 

Medium 



This picture will enable 
yoit to explain sunrise | 
sunset » and time zones* 

Motion picture of earth 
with cities labeled* 
tation of earth brings 
sunrise first to Ne«r York^ 
then to Denver# then to 
San Francisco* Narration 
and arrows illustrate sun- 
rise, noon, sunset » time 
zones* 

Movie sequences of the 
aemory device illustrating 
"Rule of the right-hand#." 
which mediates recall of 
rotation toward the east* 

Programed instruction with 
drawings of other cities on 
the globe* Responses are 
"rotates toward east"| 
correct response to order 
of sunrises in various 
cities; correct responses 
(hours) for time zones* 



Instructional Event 

1* Inform learners of 
performance required 

2* Present stimulus* 



3* Present prompt* 
4* Appraisal* 
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5. Generalizing experience. Explain "one day of tiac" in 

terms of the earthy sun^ 
rotation* and axis," 

This example* as veil as many others* are presented by 
Briggs to illustrate the application of instructional specification 
to a programed course in geography. It might interest you to 
know that the course was prepared by Br# Vincent N, Campbell in 
collaboration with science teachers in local schools in the Palo 
Alto* California area (3), The materials were prepared for an 
experiment on program-directed study versus self-directed study 
( 2 ). 



The purpose of presenting the preceeding examples is to 
illustrate how an instructor* together with an instructional 
specialist and a media specialist* night predesign instructional 
conditions. Of course* there is more to it than what has even 
been mentioned above, Se(|uences of objectives must be reviewed 
in an overall fashion to make sure thati edia choices are reason- 
able, Specifications must be prepared for the production per- 
sonnel, Informal testa of the feasibility of various sequences . 
must be run on a small nusdier of studepts whenever required. 

Only then may a prototype be developed and tried out. 

If time allows* take an instructional objective you are 
familiar with and attempt to apply the steps presented above. 
Conduct an objective analysis on that objective to discover the 
sequence of enabling objectives. Attempt to identify the 
types of learning involved for each enabling objective and to 
specify the learning conditions for each type of learning. Then 
specify the instructio^l events and the medium for presenting 
each learning experience. Where you have questions or problems* 
note these and bring them up in the aggl^ discussion 

period. If you think you have a better method of prescribing 
instructional conditions* please do not hesitate to mention 
this in your discussion groups as wall. 
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Appendix A 

Conditions for Principle Learning 

"Conditions within the Learner * The prere<;?xl8lte for ac- 
quiring the chains of concepts that constitute principles 
is knowing the concepts. Birds fly south in the w^.nter is 
easily learned as a principle tdien the learner has already 
learned all four concepts Involved in it* There is, of 
course I a kind of *partlal* learning of a principle that 
nay result when the individual knows only some of the 
coiqponent concepts. Should a learner know all the concepts 
except south, it is apparent that some kind of principle 
eottld s'Hll be learned, but it would be an inadequate one. 

As previously emphasized § knowing the concepts means being 
able to identify any members of the class they name. It is 
only when such prerequisite concepts have been mastered that 
a principle can be learned tdlth full adequacy. Otherwise, 
there is the danger that the conceptual chain, or some parts 
of it, will become merely a verbal chain, without the full 
meaning that inheres in a well-established principle. It 
is unfortunately true that inadequate principles can be 
learned. It is a challenge for instruction to avoid these, 
and it is a challenge for measurement techniques to dis- 
tinguish them from adequate ones. 

CJonditions in the Learning Situation • The ma^or external 
conditions of principle learning are embodied in verbal 
ins true tions . The example of instructions used xfith round 
things roll will be useful to recall here. 

1. The conditions of principle learning often begin with 
a statement of the general nature of the nerformanc_e ^ 
be expected when learning is complete . In the previous 
example, the Instructor says, *I want you to answer the question, 
What kinds of things rolll!* Why does he say this? Isn’t 
he simply stating the principle, giving it away, so to 
speak? The main reason for making such a statement, which 
the learner ’holds in mind’ during learning, appears to be 
this. It provides the learner with a means for obtaining 
immediate reinforcement when he has reached the terminal 
act. Having this statement for a model, he will be able to 
know when he has finished learning . and in many cases , when 
he has acquired the correct principle. Since principles 
may be long chains, the learner may need to have a conven- 
iently retained ref<^rence to tell him when the end is 
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reached* The lastructory thoughi catmot be said to be 
'telling the principle*' He doesn't state the principle 
Itself • but only the kind of performance that will demon- 
strate the attainment or the principle* 

2* Verbal Ins true tlona continue by Invoking recall of the 
component concepts* The Instructor says* 'You remember 
what roll means. • « • You remeeber what round means*' In 
many cases « the recall of these concepts is stimulated en- 
tirely by verbal means* In others (as in the example 
preyipusly given) the class of stimuli that represent the 
concept may also be shown; the student may be asked to recall 
the roll event by identifying one^ and a round thing by 
picking one out* Pictures 9 of course 9 may be used for this 
purpose as well* 

3* Verbal cues are next given for the principle as a whole* 
In our simple example 9 the verbal statement 'Round things 
roll' accomplishes this purpose* Howeverf it should be 
noted that thase verbal cues to the principle need not be 
an exact verbalization of the entire principle; they are 
in this case only because the principle is such a short 
one* If the principle were one from elementary geometry 
like 'An angle is formed by the intersection of two r^ys,' 
the verbal cues may be contained in such statements as 
'Here are two rays* They intersect* We have an angle*' 

Such statements do not correspond exactly to an acceptable 
verbal definition* Yet they function as well or better in 
providing verbal cues to stimulate the learning of the 
principle* 

4* Flnallyt a verbal question asks the student to demon- 
strate the principles The instructor says 9 'Show me*' 

The exact form is not of great importance so long as it 
truly requires the student to demonstrate the principle in 
its full senseo Added to this may be the requirement 
of asking the student to state the principle verbally 9 
as when the Instructor asks9 'What kinds of things roll?' 

But note particularly ^Jhat such verbal statement is not 
essential to the learning of the principle! nor does it 
prove the student has learned the principle* Then why Is it 
down? Probably for a very practical reasons the instructor 
wants the student to be able to talk about the principle 
later 0119 and so he teaches him the right words to say* 

This is undoubtedly useful 9 but it Is important to note 
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that this kind of verbal chaining (‘learning the definition’) 
is an unessential part of principle learning itself, 

5, The presence of some familiar learning conditions may be 
recognised in principle learning. Contiguity appears to 
be an important condition applicable to the time in terval 
between the recall of component concepts (step 2) and the 
verbal cuing of the principle with these parts properly 
sequenced (step 3) . Reinforcement io provided when the 
principle is exhibited in its complete form. The instructor 
may say, ’Right’’ Even prior to this, the student may 
receive reinforcement by matching his terminal act with a 
form remembered from the initial instruction (step 1). 
Reoetitlon has not been shown to be an important cwdltion 
for this kind of learning, nor for its retention (Gagne 
and Bassler, 1963). However, the particular finding cited 
was obtained when the principles learned occupied an isolated 
position within ether varieties of principles. The possible 
need for repetition in the form of review exercises to 
overcome the effects of interference should not be dis- 
counted. Forgetting, after ai, is a highly likely occur- 
rence with any kind of learned capability. 

The capability acquired when a principle is learned must be 
someT/jhat abstractly represented, since by its very nature it 
cannot be related to specific stimuli, but only to the 
classes of stimuli that arouse its component concepts. As 
previously stated, a simple principle like ‘birds fly 
may be considered a chain of two concepts, and diagramed 

as follows: 



Concept 1 Concept 2 




’•bird**’ 

(verbet chain) 

The terminal response, here designated as ’pointing to 
birds flying,’ may take many forms, so long as it con- 
stitutes a demonstration of the principle. Also Indicated 
is the possibility of formation of the verbal chain birds 
fly,’ ^diich is the verbal statement of the principle, not 
dn Integral part of the principle Itself ** (7, p. 146-149). 
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Evaluation She^c 



We need to Icnow how well the ideas and issues in this loanual are 
conmunlcated to you. You are the test audience for this material. 
To remove or strengthen the weak spots » to retain or improve the 
strone ones Tlf anv). an account of vour learnlne escoerlence as 
you read these sections is crucial. Use this sheet as you study 
and jot down your reactions. Don't be concerned with typographical 
errors. We're concerned with how the message is coming through. 

SECTION # TITLE 



Give topic, 
paragraph or 
sentence, page 



Importance 
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1 2 3 4 5 
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Clarity 

Rate 
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Over-all Rating of the Section: Use a 5-point Scale 
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SECTION IV 

Measurement 

H. Del Schalock 



INTRODUCTION 

Unlike the ocher topics covered In the Institute, aeasure&ent 
plays a central role in both instructional systems development and 
Instructional syst^ research. With respect to instructional sy8**> 
terns development, as Dr. Hamreos has pointed out, measurement 
serves three functions: 1) the determination of Whether a learner 

achieves the level of performance that is expected of him, l.e«, 
Whether the terminal objectives and their various enabling obJec-> 
tlves have been realized, 2) analysis of an inatructional system 
to determine those aspects of it that are effective and those that 
are not, and 3) evaluation of the appropriateness of the concep- 
tual framework that underlies the instructional system. This 
involves the re-analysis of the relevance and ordering of enabling 
analysis 2. From this point of view, measuremen t in instructional 
systems development is much more than simply "evaluating stud^t 
performance;" it Is an integral, inseparable, absolutely c>;nicial 
part of the entire process. Because of this, the quality of 
instructional systems is closely related to the qu^lty of meas- 
urement that is associated with them. 

heasurement plays an equally critical role in instructional 
systems research.^ It Is also an expanded role. In instructional 



1 

'^For purposes of the present paper the term instructional 
systems evaluation, or simply evaluation, 'i$ill be used to refer 
to the various activities in the over-all task of instructional 
systems development which depend upon measurement. The term 
instructional systems research, or simply instructional research, 
will be UEsed to refer to a set of activities that are considerably 
different than those involved in evalxiation. Evaluation is a 
relatively limited, narrow range of activities that lead to infor- 
mation which has relevance only to a specific instructional system 
within a specific context. It does not lead to principles, gener- 
alizations, laws, or theory. Research, on the other hand, refers 
to the much broader class of activities that lead to the extension 
of generalizable knowledge. Research that relates to instructional 
systems focuses upon such matters as the appropriateness of an 
Instructional system for various kinds of children, the relative 
effectiveness of altematl\*e media forms or presentation strategies 
within a given Instructional system, the generalizabll.lty of an 
Instructional system across subject matter areas, etc. 



s^tans development, meeauremont focusee ona.y upon the oMectlvca 

iaatructlonal research, it m2dLnt&ln8 this 
-ocus, -ut includes ^ athers as well. In their nost general 
terms, these are: 1) descriptions of the treatment conditiotiii 

the system, the form 

^5 appears, and the methods hy shich It Is 
presented, and 2) descriptions of the contlnge ngir factors within 
JM..setting which interac t with the ctr^toaS SL^?ions! 

thi^ kinds: a) learner characteristics, c.go, 

experience, b) teacher charac- 

n? '^^clfSround of knowledge, training, IntereBts, and 

«wsiijthJif«^ number of children in a class, 

S rt is thsse factors, 

* '**^®*‘ determine whether an instructional system wUl be 

far ^ controlled as 

^ all instrjwtional research. In fact, without the 

umttivi V aoatrol of these factors, instructlon<!tl research 

instructional evaluation would not 
be possible without the assessment of Instructional objectives. 

^ isstructlottal evaluation end instructional 

are completely dependent upon measurement. Without it, 

J« 4* neither. But the issue here is not one of whether there 

^ that measurement has 

is ^2®® activitieg. The real issue is whether there 

to ffi^suesmeat, and most educators are not able 

distinction. Broadly speaking, the purpose of the 
present chapter is to help them be able to do so. S^lflcallv 

*.2! 2 2 maasurement in evaluation and research, b) provide 
for him fche information needed to wisely select the formal, exist- 

®®® ^ ^^® 8^8^ nation and research 
hit-iia 4.k* to develop in him the competencies needed to 

high quality informal, '’teacher*made" measures that he 

needL^^to'h.Hi? . Developing the competencies 

build formal, atandardlsed measures r is beyond the pale 
of the present effort. 

^JPalj^ly* the chapter also will clarify the nature of 
a^^tlonal and psychological measurement, for by-and-large it 

^8 ^«d to understa^ or 

Pt^alplaa involved in some of the more common measure- 

S vol^ sciences, for example, length, weight, 

ana volume, it is herd sometimes to understand and accept the 



fact that the measurement of such characteristics of indivi- 
duals as achievement, intelligence, aggressiveness, and anxiety 
involves basically and essentially the same thinking and the 
same general procedures t It is also hard to accept the fact 
if TuXss of iQOEsuiroiziontL esn sst \ip on 8or?i® 
or empirical basis, measurement of anything is theoretically 
possible. From the point of view of the writer it is Important 
that educators somehow develop this point of view for with it 
they will not reject the possibility of measuring some property 
simply because it is cosj^^lex and elusive. They will, as Gulon 
has put it ” . . . understand that measurement is a game that th^ 
may or may not be able to play with this or that property at 
this time. But they will never reject the possibility of playing 
the game, though they may realistically understand its difficulties”. 
(1966, p ). 
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A POINT OF VIEW TOWAI® MEASOREMENT IN EDUCATION 



In addition to tacognlzlng the place of measurement in 

ei^luatlon and research, and being comltted to the 
idea that that which needs to be measured can be, the educator 
^t also be aware of some of the realities that are involved 
in educational measurement. In this writer’s opinion fcur such 
realities" are paramount. 

, . educator or education al researcher m uat understand the 

jimd^ntal nature of mea s urement, particularly as it annlle a 
^ehavioral scienc e s, if he is to Pursue evaliiatlon an d 
,gsearch act iyi ties .j udicious Iv . Simplified guides as to when 
to jwe ^at kinds of measures upon whom are not enough. If edu- 
cation is to be effective, and if a "science" of education is 
to erorge, the nature of educational measurement must be under- 
stood by those who use it. Lindquist states the view well when 
he says If measurement is to continue to play an increasingly 
import wt role in education, measurement workers must be much 
more than technicians. Unless their efforts are directed by a 
sound educational philosophy, unless they accept and welcome a 
greater share of responsibility for the selection and clarifi- 
cation of ^ucational objectives, unless they show much more con- 

4 measure as well as how they measure it, much 

of their work will prove futile and ineffective." (1950, p 158). 

Toward this end, an Appendix is attached to the present 
chapter (see Appendix A) which Introduces the reader to the con- 
cept of measurement generally and to the characteristics of measure, 
ment in the behavioral sciences specifically. While it is not 
cssumed that all will read this material during the course of the 

iMtltute, the reader is encouraged to become familiar with it as 
tune permits# 




f6resice8 9 moral valiueSj, attitudes toward socd.al Institutions 
and practices, habits relating to personal hj^glene ar . physical 
fitness, managerial or leadership ability, etc* have been seriously 
neglected— ’even though most educators hold them to be of impor- 
tance. The point of view adopted here is thait if these 
important classes of objectives, they should be so specified and 
instructional systemt developed to bring them about. In line 
with earl'^'^r comments, this would require that appropiriiite measures 
of the'' jectives be established. 

3) If the educator or educational resegircher doc^fi uad<irtake 
the measurement of ihe full range of edu<;atlonal obj ective£> to 
which he Is committed, he will have to rely unon more thiM i ”pape_r 
*nd pencil** measures . This point of view rests upon the afjsump— 
tion that the aim of any educational achievement measure is to 
obtain a sample of the criterion behavior, i.e«, the behavior which 
is specified in the behavioral objective, or a sample of be- 
havior or the products of behavior which presumably relate to 
the criterion behavior. How else can one obtain evidence that 
an (Educational objective has in fact been realised? Given this 
point of view, paper and pencil tests are not at all appropriate 
to the measurement of the full range of educational objectives. 

When the focus of measurement is upon ’Icnowledge*', e.g., concepts, 
facts, and principles, or upon the application of knowledge to 
a set of tasks that require only the manipulation of symbols, 
e.g., the solution of a mathematics problem, outlining the steps 
Involved in building a house, or writing a theme, on the expression 
of consideration in one's relations with others, paper and pencil 
measures are perfectly appropriate. However, when the focus of 
measurement is upon building a house, or upon relating considerately 
to others, a paper jand pencil measure won't do— unless cine is 
willing to make the assumption that being able to outline how to 
build a house or how to behave considerately is in fact related 
to the ability to build a house or to act considerately.. While 
most people would probabl'^ accept the idea that the knoifledge 
factor is related to the concrete performance factor, few would 
accept the Idea that the relationship is perfect. If this is 
true, than measures other than or in addition to those which re- 
quire only the manipulation of symbols are needed in order to 
assess some of the objectives of education. 

Toward this end it is proposed that the educator or educa- 
tional researcher must be familiar with the full range of mea- 
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sures available to the behatioraX scientist* Tliese include such 
"obtrusive" ueasures as interviews, standardized objective tests, 
teacher-nade tests t. systematic observation (facc'-to-face obser- 

VS»W«iVM| f O**# -Si 

projective tests, and such "unobtrusive" measures as physical 
traces through erosion and accretion, e*g*, wear on library boohs 
or the accumulation of ^ised paint tubes, docunen\fa and products, 
simple observations, and contrived observations thxioiigh hidden 
hardware* Sach of these ate specialized meaaureiiient methodologies, 
acme of i<ihich are a;pprop>riate to one kind of mei?u!iut(3iDent problem 
and 8omfi^ appropriate to otiiers, but in order to haodle the full 
range oHf measurement tasks thiit he faces an educiit<»r or educa* 
tlonal resear<sher needs to be competent in them al.!* 

Idmitationa of space <Tsnd time make it impractical, however, 
to enter into a discussion of the various methodologies in the 
present paper* They are complex and each has its own unique 
set of problems * To provi de the reader with some idea as t® tihe 
nature of l^mae measures, however; the ld.nd of data that they 
provide, and some of the particular strengths and weaknesses of 
each, the major classes of measurement methodologies are suamarized 
in Appendix B. While this brief summary will in no way prepare 
the reader to use the various classes of measures hopefully it 
will help sensitize him to the possibility of their use* Excellent 
discussions of the operations involved in the various measures 
appear in Eestinger and Katz (1953) > Llndzey (1954), Mussen (I960). 
Gage (1963), Kerllnger (1963), and Webb, et al* (1966). 

4) Because of the "inexact" nature of measurement in the 
behavioral sciences (see Appendix A) educators or educational 
researchers should bring several different claasea of measures, 
to bear upon each behavioral objective that is to be assessed* 

There are two reasons which underlie this recommendation: 1) any 
single measure c£in hope to assess only one or a few of the many 
enabling objectifies which comprise a terminal objectives, (or 
perhaps even only a few of the properties which define an enabling 
objective!) and these only within a limited saoiple of the situa- 
tions in which the enabling objective is reflected, and 2) 
because all measures, whethei: in the physical or behavioral 
sciences. Involve s degree of error* This is especially the case 
in education, tiowevex, where many of the measures must of neces- 
sity be indirect measures, i*e*, they are indicants of or supposed- 
ly related in some wav to a given educational outcome* Given 
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th686 two conditions the educator and educational researcher need 
to employ what Campbell and Fiske (1959} have called a "multiple 
operations*' approach to measurement* In brief* it calls for 
multiple measures to be used in every study, and assumes that in 

V 4 *.1.^ *.*<111 <ln <*riA ifAlAVAnt: comoouents 
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(of that which is to be measured) but have different patterns 
of Irrelevant components" (Webb et al*i> P* 3)* The basic assump- 
tion underlying the procedure is, simply, that once a proposition 
has been confirmed by two or more independent measurement processes 
the uncertainty of its interpretation is greatly reduced. Camp- 
bell and Fiske argue that the most persuasive evidence for the 
existence of that which is being measured comes throu^ a triapgit^ 
lation of measurement processes* The basic assumption underlying 
the approach is that if a proposition can survive the onslau^t 
of a series of Imperfect measures, with all their irrelevant 
error, confidence can be placed in it. 



A sample list ing of multiple criterion measures by class 
of educational outcome appears as Ap>pendlx C« 









MEASUREMENT IN INSTRUCTIONAL SYSTEldS DEVELOPMENT 



T^o fl iTSSlisr wjLds T&ngs of SwwSldS^dlLzod 3Ci^iSV0!!lSXlt 
measures exist (see Buros, 1965) » most measures that are used in 
the evaluation of instructional systems are of the informal, 
teacher«-made variety. Considering the almost limitless number 
of educational objectives that are open to selection, and the part 
that individual teachers play in their selection, this probably 
will continue to be the case for many years to come. Since these 
measiures are nearly always specific to each instructional system, 
and therefore need to be developed by the person who is developing 
the system, the focus of the present section of the p^iper is 
upon the development of this kind of measure. 

Within this focus five topics are considered: 1) planning 

title measure, 2) developing the measure, 3) trying out the measure. 

4 ) evaluating the measure, and 5) the "formal” attrlbucco to be 
csnsldered in judging the worth of an informal measure. The 
procedures and attributes reviewed within these topics are appll- 
Ciible generally across all classes of measures within the be~ 
havioral sciences (see Appendix B), that is, they are not limited 
only to paper and pencil measures. However, for persons wishing 
concrete, specific guidance in the development of paper and pencil 
classroom tests five recently published books provide excellent 
references: Sax, G. The Construction and Analysis of Educational 

md Psychological Tests; A Laboratory Manual (1962); Ahmann, J.S. 
and Crock, M.D., Evaluating Pupil Growth (1963); Stanley, J.C. 
Measurement in Today *s Sdiools (1964); Gronlund, N.E. Measurement 
and Evaluation in Teaching (1965); and Ebel, R.L. Measuring Fdiuca- 
tional Achievement (1965). The pros and cons of various item 
types. Including essay items, how these can best be developed, 
how to administer, score and use tests, etc. are discussed in de>- 
tail. An article by Eva Bake., on the development of prototype 
items is also of valve in this respect (see Institute Materials). 

Planing the Measure 

As Dr. Hamreus has indicated, one should begin the develop- 
ment of the measure to be used in assessing criterion performance 
as soon as the hierarchy of enabling objectives has been identified. 
There are two reasons for this: 1) it eliminates the pitfall of 
assessing; "that which has been taught" in contrast to "that which 





was 8uppo8®d to feavs been taught*' > l•e«^ the behavioral objec— 
tivee,^ md 2) In the process of building a measure to assess 
the varleos enabling objectives clarification may come in the 
en^llfttg objectives themselves. Given a hierarchy of enabling 
objectivea there are three taSKs to be accO^lished In the test 
planning phase: 1) a decision as to the fidelity of the items to 

be used ii!, the measures that 18$ the extent to which the response 
to itcsms within the erasure will require the performance of the 
conerete behavior specified in the behavioral objective, in con- 
trast to the performance of behaviors which are only related to 
it, e.g., buildirag a liuUse its. describing how cne would be built 
or acting considerately vs. describing what acts of consideration 
involve^, 2) a decision as to the rem r esentativeness of the Iteaas 
to be t;»ed in the measure, that is, the number and range of situa- 
tions tc be sampled by the items in which the behavior is reflect- 
ai, e»g., does one ask to have one house of one kind built or 
three houses of ons icind built, or ones house of threci differtmt 
kinds built (to tec<t for craaietfer?) , and 3) the weigtit to be given 
to each item. The question of fidelity, of course, iirevolvee 
around the issue of inference: is one filing to accept as evi- 

(ience of the accomplishment of a terminal objective something less 
than the acttial performance of that objective? Is one willing to 
accept a behavior supposedly related to the terminal objective 
as evidence of the objective, and if so how doss or how distal 
can the relationship be and still be acceptable? For example, 
is one wi.lllng to accept as evidence of conslderatsness toward 
parents a. verbal statement to the effect that the examinee i£i con- 
siderate? Would observation of considerateness toward animals 
or children be any better? Would considerateness toward peers or 



2to eliminate a reverse kind of pitfall, there is a gen- 
erally agreed upon rule in educational, measuiment that it is not 
cricket to focus an instructional system upon an item that is 
Included in the evaluation measure. 

^It may be of course that the "behavioral objective" Itself 
involves only the description of how to build a house w In this 
sense, the issue of fidelity rests initially with the instructional 
systems builder. Even so, it is a3.so always an issue in the devel- 
opment of measures to assess a behavioral objective, for the cir- 
cumstances of an educational setting may force the test developer 
to settle for a level of fidelity that Is less than desired. 









a teacher be better still? These are the kinds of issues in- 
volved in the question of fidelity, and they are crucial in the 
process of achievement measurement. 

The question of representativeness revolves around the issue 
of generalizabllity : how many and what kind of situations must 

be sampled or observed in order to feel confident that the cri*" 
terion behavior is sufficiently well established that it is 
a functional part of the behavioral repertoire of the examinee? 

The question of item weighting is simply a matter of deciding 
whether some items will be more informative of criterion perfor- 
mance than others. 

Unfortunately, there are no firmly established rules to 
guide decisions on any of these three questions. The level of 
fidelity desired in an instrument is a matter of pe^pn^l pre- 
ference (though one probably should he able to defend it) inter- 
acting with the reality demands of a situation. It is sometimes 
difficult to set up a testing situation which h«» ttehlfieeturai 
stunts building houses. The extent of evidence required as to 
the representativeness of the pcrforaiancc measure is also largely 
a matter of personal preference and reality demands: generally 

speaking, the irp44ei? the sampling of items the better the evi- 
dence, but also the hisber the cost. Establishing rules for 
weighting item responses is also arbitrary, though in most short 
ans'ijer tests of educational achievement equal weight is given 
to all items. In measures involving essay questions, or products, 
or interpersonal hl^havior the matter of assigning item weights 
is equally arbitraiy. The single rule that could be thou^t to 
operate in all these decisions is that of logic: the fidelity 
of items and the representativeness of situations sampled and the 
assigning of item weights must somehow m<*" ^he demands of elemen- 
tary logic. If a decision on any of the$ :tters wrenches one s 
credulity, it is likely that even this ie criterion has been 
violated. 

The lack of established rules for making planning level 
decisions should not be talcen to mean that these are insigni- 
ficant decisions. Indeed, of all the decisions made in relation 
to measurement they are perhaps the most basic, for all else 
stems from them. In order that the reader get a firm grasp of 
the concepts involved in these decisions the topics of fidelity 
and representativeness are reviewed again under the heading 
Formal Attributes of ILnformal Measures. 




DeveloDinjt the Measure 

Given the enabling objectives that are to be measured , and 
clarity as to the fidelity of items and the range of situations 
to be sampled by those items » one is then ready to proceed tilth 
the development of the measure. Three steps are involved in this 
process: 1) the specification of the set of operations that are 

to be taad la assessing the status or condition of each of the 
objectives that are to be measured, (the development of an **item” 
pool), 2) the specification of the eocditicns under ^diich these 
operations are to be made (the development of a set of directions) 
and 3) the specification of the rules by Which numerals are to 
be assigned to these operations (the development of a "scoring** 
system). In deciding upon the set of operations to use (the 
**item types") it is necessary to look to the full range of measure- 
ment methodologies avai.lable to the behavioral scientist (see 
Appendix B). In so doing the ailm is to find the methodology that 
provides the most appropriate set of operations for the assess- 
ment of a given objectiLve at a given level of fidelity. Thus, 
if the objective being measured is considerateness toward others, 
and the level of fidelity is set at **ldent1lcal elements", i.e., 
it requires the perforiaance of the concretes t^%aviors specified 
in the objective, the measurement methodology most approp:rlate 
to the task would probably be systematic observation. Other appro- 
priate methodologies voyld Include the interview, simple obser- 
vation and contrived observation. Perhaps the leemt appropriate 
methodology would be that involving teacher-made paper and pencil 
tests! Examples of "item types** when using systematic observa- 
tional procedures might include the recording of all instances 
of helping another when help was sought, helping another idien help 
was not sou^t, "surprising" another with unexpected gifts, 
thoughtfulness, etc< 

The selection of a methodology, and the various '*ltem types'* 
within it, represents the first Important consideration in item 
development. Once this is done it can be combined with the speci- 
fication of situations to be sampled by the items, e.g., the con- 
siderateness of the examinee at home with her mother, her younger 
brother and her father and her considerateness at school with her 
"three best friends**, and the development of items can got under- 
way. Generally speaking, a measure should contain as many itaos 
as is practically possible, for the wider the sample of items 
within situations the more representative the responses to the 
measure should be. Ordinarily, it is wise to develop a larger 
pool of items than ultimately will be needed. 
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In combination, the selection of an appropriate measurement 
methodology and the selection of appropriate Item types within It 
provide a basis for assessing the relevance of a measure to 
that which Is to be measured. (For those familiar with test 
theory, the concept of relevance is comparable' to the earlier 
concepts of face or content validity— ‘See Standards. 1966). As 
was the case with planning decisions, the single rule that may 
be thought of as operating in development decielons is that of 
logic: If a decision as to measurement methodology or item type 

stretches one’s credulity as they pertain to a particular ob“ 
j active then it is likely that the criterion of logic has been 
violated. 

Since paper and pencil tests are widely used by educators, 
and are appropriate for the mea< 3 urement of of ahs educational 
objectives in the cognitive domain, special attention will now 
be given their development. In entering this discussion it is 
to be realized that all of the planning and development decisions 
that have been reviewed thus far apply to the development of 
paper pencil measures . 

As with any oidier methodology it is wise in developing paper 
and pencil measures to develop a larger pool of items than 
ultimately will be needed. Also, it is wise to employ more than 
one type of item, e.g., completion, true-false, multiple choice, 
ranking. Items should be phrased so that the content of the 
statement rather than its form will determine the answer, that 
is, avoid telltale words such as "always”, "never", "entirely", 
or "absolutely", which usually lead to an answer being false, 
or such words as "may", "sometimes", and "as a rule" which are 
most often associated with true statements. All items of a 
particular type should be placed together in the test. 

A parl:icularly critical issue for tine builder of classroom 
paper and pencil testa is that of item difficulty. Obviously, if 
an item is so easy that everyone taking the test answers it 
correctly, then it is of no value In discriminating bem those 
who have more and those who have less of the property I . mea- 
sured. The same problem exists if an item is so hard tuat no 
one answers it correctly. Yet for purposes of moralr^ the re- 
duction of test anxiety, etc. it majr be good policy to put 
in a few Items at the beginning of a test that are extremely easy, 
and for purposes of maximum discriminatory power it may be de- 
sireable to put In a few items at the end of the test that few are 



likely to get right. As yet there are no agreed-upon rules to 
govern treatment of this dilemma » but a coononly appearing rule 
of thumb suggests that most items in the test should be of approxi- 
mately 50 percent difficulty, that is, approximately half of the 
group being tested should knew the answer. This rule evolves 
from experimental work which shows that under these conditions 
a test has in ayimu m discriminative power. In practice, however, 
test makers generally produce test items having a wide range of 
difficulty with only the average level of difficulty being 
approximately 50 (Guion, 1966, p. 200). 

In advance of giving a test one can never be sure, of course, 
how difficult a particular item actually is. A procedure for 
determining item difficulty is outlined in the section on 
evaluating a measure. 

After the item pool has been established, the two remaining 
Instrument development tasks can be undertaken, namely, the 
specification of the conditions \inder which the items are to be 
administered .and the specification of the rules by which numerals 
are to be assigned to that which is assessed by an item. In 
the example where measurement Involved the observation of a girl 
relating to others , specification of the conditions under which 
the items are to be administered might include such statements as 
"after dinner alone with her mother in the kitchen” or "at bed- 
time with her brother upstairs." Directions for taking a paper 
and pencil test also must be clearly stated. In general, ^in- 
structions for taking a test should be so qI&Cx that the least 
able student in the class knows ^at he is expected to do, even 
though he can’t do it. 

Rules for assigning numerals to that which is being 
assessed ("scoring" rules) vary widely with measurement metho- 
dology, item type, and weighting decisions. For example, when 
preconceived category sets are used in systematic observation pro- 
cedures the scoring rule is usually 1 or 0, that is, a category 
appeared or it didn’t. When rating scales are used the scoring 
rules may be something to the effect that "when the behavior 
in question occurs frequently, check scale position 3, when it 
occurs seldom or not at all check position 1; when it occurs at 
a frequency somewhere between 3 and 1, check position 2." 

Most short answer paper and pencil tests also provide a score 
of 1 or 0, although some involve a 41 or a -1 with the -1 serv- 
ing as a penalty for guessing. Formulas do exist that permit 
correction of "chance" or "guessing" on short-answer paper and 
pencil measures (Guion, 1966) . 



Trying cut the Measure ^ 

Since it Is impossible in ndvance to know how good a measure 

ifl _ ny Imriu j gOOd STl^ SS'S poor 9 S SryOUt 

of the measure is essential* IWo rules ncsed to govern the 
tryout: 1) every reasonable precaution should be taken to in- 

sure the best of measurement conditions » and 2) the time allowance 
for obtaining the measure should be generous* Since both rules . 
mean somewhat different things for different measurement metho- 
dologies, e.g., in systematic observation the setting being 
observed needs to be "natural”, attention is to be directed to 
the normal course of events, and observer influence needs to be 
at a minimum and in paper and pencil testing the situation needs 
tc be quiet, attention has to bc^ directed to a specific task and 
teacher influence (presence) needs to be high, administration and 
timing rules will not be discussed here. For those interested 
in administration, timing, etc. in relation to most of the metho- 
dologies in the behavioral sciences see Festlnger and Katz (1953), 
Lindzey (1954), Mussen (1960), and Rerlinger (1965) o For those 
Interested in the administration of paper and pencil measures in 
the classroom see Stanley (1964) apd Ebel (1965). 

A good deal can be learned ^oiit a measure from simply 
att^dlng to the response of examinees to it. For example, in 
an interview situation one can obtain a fair idea as to which items 
are clear and «diich are confusing, or whi<di are probing "sensi- 
tive" areas and which are not simply by observing the respondent's 
reaction to the items. Similarly with a paper and pencil measure: 
the overt reactions of students to the test will often suggest 
Aether it is understandable or not, whether it is hard or easy, 
whether it is relevant to the objectives of the class or irrelevant 
to them, whether there is too much or too little time, etc. 

While a measure will not stand or fall upon Information as infor- 
mally obtained as this, it is nevertheless information that is worth 
obtaining. 



Trying out a measure before it is to be used for evaluation 
purposes is b.ased upon the assumption that a measure is to be 
developed that will be used more than once* Xf this is not the 
case the teiit should be read or the measure tried by at least one 
other person before it is administered foinoally. 




Evalua^ine the Measure 



In addition ta the Informal assessment of a mgagityg^ 22 
described above, a formal assessment may be made once the measure 
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of the target audience specified for the measure* The na- 
ture of the evaluation depends, however, upon the use to which the 
measure is to be put. For example. If a test is to determine 
mastery: to ac absolute criterion and the criterion involves pass- 
ing all items, then one loollcs only at the number of items answer- 
ed correctly^ if it la to serve a diagnostic function, then 
interest is not so much in how many but in tdiich items were 
passed or falledo If a test is to distinguish between students 
on that ^ilch is being measured, such as academic achievement, in- 
telligence, etc., then one looks at each test item in terms 
of its ability to do so. Since measures taken in the classroom 
most c<Hnmonly serve this last function, attention will be directed 
in the present paper to it. For reasons of space, the discus- 
sion will take as a point of focus only a short-item paper and 
pencil measure of achievement. The discussion can be genera- 
lized, hoKCVer, to the evaluation of all measurement methodologies. 

In evaluating a test for its effectiveness in discriminating 
betweem students, the range and distribution of the totiil test 
score is only a rough indication of :lts adequacy. Since a test 
is made up of items, a total test score Is dependent upim the 
discriminating power of those items; if only part of the Items 
are discriminating then the test score is reflecting the «rork 
of only those items. It is possible, through Item analysis 
procedures, to determine which Itcons are. and are not dlsci'iminat- 
ing. The value of an item aiialysis is that It permits one to know 
which items to eliminate in the next edition of the test *ind how 
many new items to add. (Obviously, in using this procedure, an 
item analysis has to be made each time new items are added) • 

By following item analysis procedures one can continuously in- 



5 

A teacher must also decide '^diether to evaluate students in 
terms of an absolute criterion, that is, whether to evaluate 
students agalniit a sta^adard which the teacher or somecne else 
sets or to evaluate tb,Bm in terns of their performance relative 
to the performance of others. A recent paper by Gerlsck., Schutz, 
and Bifker (see Institute Materials) provides an excellent dis- 
cussion of this issue,, 
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crease the power of a test to dlscrtmlfidte» as velX as increase 
one’s confttlence that individual items do in fact discriminate 
across various class groups* 

The simplest procedure for testing the discriminating 
power of an item is to determine the number of correct responses 
to the item by the students who rank in the highest 27 percent 
of the class on the teat as a ^ole. and to compare this with the 
corresponding number in the lowest 27 percent of the class* The 
items for which the number of correct responses of the high group 
most exceeds that of the low group are most discriminating; 
those In ^ich the number of correct responses of the high group 
falls behind that of the low group, and those in which the 
numbers are the same, are not discriminating* These are the items 
that should be rewritten or discarded* The basic assumption 
underlying this procedure it that the initial total test score 
is a relatively accurate measure of the behavior under considera- 
tion for all items subseouentlv kept and added are based upon that 
score* To many, this is a difficult assumption to accept* 

Formal Attributes of Informal Measures 

In the preceeding sections of the paper attention was 
directed to tliree concepts: fidelity, repreuentatl^^eness and 

relevance* Wlille these properties or atK*’ibutes of a test were 
dealt with In liome detail within the context of test: develop- 
ment, they are of sufficient Importance to be reviewed a secend 
time* Also, to be utiderstood fully, they need to be seen within 
the context of other test properties, for example, reliability, 
validity, accuracy, etc* The purpose of the present section of 
the paper is to set these various concepts in this kind of perspec 
tlve* 



A basic assumption underlying the present discussion iJii that 
measures of ciducational objectives or edu.'.ational adilevemeiat 
are direct measures, that is, they assess specified behavioral 
objectives* <md in this sense they differ from many other be- 
havioral sci(stice measures that are of necessity indirect , e*g*, 
measures of intelligence, anxiety, authoritarianism, etc* In 
the cose of aiisessing educational objectives the t.ask is to 
obtain Bieasures of ithe behavior in question, or meaaiures which 
are believed to be related to it; in the case of assessing such 
eharacterlstdcs as intelligence or anxiety, which in themselves , 
are Inferred characteristics or constructs (see Appendix A) 



the task is to obtain measures of Indicants of the characteristic 
or construct* Mille the concepts of relevance* fidelity* re- 
presentativeness* reliability and accuracy are appropriate to 
both direct and Indirect measures . it is only in the fil ^jp 
Indirect measurement that the notions of operational definitions 
construct validity come to bear. The concept of predictive 
validity is also applicable to both direct and indirect measures* 
but It becomes an Issue only when prediction of some form or 
another la involved (see Standards* 1966). Both the concept of 
construct validity and predictive validity will be discussed In 
the section on MEASUREMENT IN INSTRUCTIONAL SYSTEMS RESEARCH. 

The discussion which follows has as its focus only the 
attributes of direct measures* in this case the attributes of 
measures which are aimed at the assessment of educational out- 
comes* The intent of the discussion is to provide a sumoary of 
the attributes needed by measures of this kind before they can 
be used with confidence. 

Releva nce . As indicated earlier* the relevance of a measure 
refers to the extent to which it is a logically appropriate mea- 
sure for that which is to be measured. Technically* the relevance 
of a measure is determined by two considerations* the selection 
of an appropriate measurement methodology and the selection of 
appropriate item types to be used within that methodology. If 
either selection stretches credulity, then the relevance of the 
test is suspect. As used here* the concept of relevance is in- 
tended to replace the earlier concepts of face validity and con- 
tent validity (see Standards* 1966). In terms of the concept 
of error variance (see Appendix A) relevance is one means of 
reducing constant error* 

Representativeness . The representativeness of a measure 
refers to the extent to which it sanples the situations in which 
the behavior under consideration is reflected. Generally speaking* 
the greater the number of situations sampled the more confidence 
one can have In judging whether the behavior observed is or 
is not a functional part of the behavioral repertoire of the 
individual. As in the case of relevance* the primary means for 
determining the representativeness of a measure is throu^ logi- 
cal analysis, if the situations sampled are not representative 
of the full range of situations available for sampling then the 
measure is suspect. A formal means for estimating representative- 
ness also exists; namely, the comparison of scores obtained on 
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two forma of the neature. If scores are sinllar on both foms 
then It nay be assumed (granting the acceptability of the measure 
by logical analysis) that the sanpllng of situations Is adequate. 
The two forms of the neasure may be developed separately or they 
may be formed simply by randomly dividing the ^Itujatlons sampled 
Into two equal halves • This procedure will be recognised as 
"alternate fora" and **spllt«half " reliability measurement; but as 
used here the concept of represntatlveness Is Intended to re* 
place these earlier concepts. In terms of the concept of error 
variance representativeness Is also a means of reducing con- 
stant error. 

Kdelitv. As used In the present paper the fidelity of a 
measure refers to the extent to which a measure calls for 
the perfonsance of the concrete behavior specified by a be- 
havioral objective; It may require behavior that Is iscmorphlc 
to the objective (identical elements) or It siay call for be- 
havior that is only In some way related to It. Ideally, all 
measures of educational objectives should be Isomorphic, but 
since the realities of educational settings sometimes make this 
Impractical the general rule to be followed Is to make one's 
measures as high In fidelity as Is practically possible. 

Many years ago Lindquist (1950) proposed four alternative 
approaches to the measurement of educational obj actives: 1) give 
the examinee occasion to do some of the things that are specified 
by the objective (an Isomorphic situational response test); 2) 
give the examinee occasion to do things similar to some of those 
specified by the objective, (a "related behavior" situational 
response test); 3) describe a situation In which the exasdnee 
would have occasion to do what the objective specifies, and then 
ask him to tell ^diat he would do In this situation or how he 
would do It (a "verbalized behavior" situational response teat); 
and 4) discover whether or not the examinee knows the facts, 
rules, principles, etc., that are presumably essential or conducive 
to the desired behavior, (a "knowledge" test) (p. 146). At 
the time Lindquist wrote his paper educators were measuring 
essentially et the knowledge level, and his plea was to get 
them to move to a higher level of fidelity. Today they are 
still measuring essentially at the knowledge level, and the 
plea is still relevant. This Is not meant to Imply that measure- 
ment should avoid focusing at the knowledge level; Indeed, 
many educatlcmal objectives are focused cmtlrely at that level. 
Also, there is obvious truth In the argument that knowledge is 
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essential or conducive to the overt behavior wJlth which an 
ultimate educational objectives is concerned, ;l. <>e. there is a re» 
lationship between what and how much an IndlviUlual knows and 
how he will behave in certain situations. Thai point of the dis- 
cussion here, however. Is that whlile the toeasiiixement of know- 

loHflrO '!«% fV|cs*nv q 1 a 

stltute for the situational response type measures as measures 
of educational objectives. 

To date the concept of the fidelity of a measure has not 
appeared in the literature on test theory, so the concept has 
had little empirical testing. On the surface, however, it appears 
to be a useful concept and so it has been included here. In 
terms of the concept of error variance it too contributes to the 
reduction of constant error. 

Taken together, the three measures of relevance, repre- 
sentativeness and fidelity in direct measurement take the place 
of the concept of validity in indirect measurement. 

Reliability. Reliability is that attribute of a test which 
speaks to the consistency with which it measures that which it 
measures. As discussed earlier all measurement, whether in the 
physical or the behavioral sciences, contains a certain amount 
of chance or random error. XWo sets of measurements of the same 
characteristics or properties of the same individuals will never 
exactly duplicate each other. This is termed the unreliability 
of measurement. At the same time, however, repeated measurements 
of a property within an Individual will (if the measure is at 
all appropriate), show some consistency . For example, if a boy 
was the best reader in the room the first time that a class was 
tested it is highly probable that he will be among the best readers 
on another testing, even though he may not be the best. This 
tendency toward consistency for a repeated set of measures is 
what is known geneally as reliability. Technically, relia* 
billty may be defined as the extent to which a set of measurements 
Is free from variance due to random error, but in fact it tends 
to be more a measure of the consistency of the individiuA that 
is being measured than it is a measure of the test per se. 

A number of factors may contribute to a lowering of the 
reliability of an Instrument: a) response variation by the 

subject due to fatigue. Illness, an "incorrect set", etc., b) 
variation 1ft adelntsteation or the administrator of the test. 
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c) variations in scoring; , etc. As long as the sources of error 
remain unsystematic, however, i.e,, not constant, their threat 
to measurement is not great. 

Within this framework there is essentially only one proce- 
dure for obtaining an estimate of the reliability of a measure, 
namely, retesting an individual or set of individuals with the 
same test. While there are a number of limitations inherent in 
test-retest reliability, for example If one waits very long 
between measures individuals will change so much that the 
repeated measure will of necessity be quite different from the 
first, or if one retests too quickly the second score will be 
subject to recall error from the first. It Is still the one 
measure that provides data relevant to the concept of relia- 
bility. Reliability estimates are needed for all measures, whether 
direct or indirect, for random error always enters measurement. 

A special problem Is encountered In reliability estimation 
when the measurement methodology being used does not call for 
the rigid control of test stimulus materials, e.g., when the 
measure is the description (categorization) of the free play of 
children, teacher-child or parent-child interaction, group 
problem solving behavior, etc. Here the stimulus situation is 
never the same twice (unless one Is working in a controlled, 
experimental situation) and consequently there is no reason to 
assume that the behavior in question will ever be the same twice. 
Nevertheless, it is still possible to obtain an estimate of the 
consistency of these kinds of measures if one is Interested in 
doing so. The only requirement is that the situation that is 
being observed be as similar as possible on repeated measurement 
occasions . e.g,, observing the same children in the same play 
area at the same time of the day under as many of the same con- 
ditions as possible. 

Accuracy. The accuracy of a measure has often been confused 
with reliability, but they are in fact two quite different con- 
cepts. A test may be quite reliable (a person may consistently 
receive a similar score upon repeated measures) but it may not be 
accurate, that is, the test could show him with x amount of a 
characteristic when In fact he has an x + 1 amount. Nor is 
accuracy comparable to relevance, representativeness, fidelity 
or validity. A test may be valid etc., that is, it tests x when 
it is supposed to test x, but it may be so gross as not to be able 
to distlnqulsh between x and x + 1. 




From a technical point of view, the concept of accuracy 
has no legitimate meaning in the behavioral sciences. This is 
because these sciences have access only to nominal and ordinal 
level data (see Appendix A); one must have an absolute measure 
against which to Judge (sudi as lose which exist in the Bureau 
of Standards) in order to have a measure of accuracy. This first 
of all requires measurement at the ratio level. 

Practicality, wtiile practical considerations can never Justify 
the use of a test which gives worthless information, a relevant 
and technically sound test cannot be used where it is impractical. 
Thus users of tests need always to seek a viable **trade*o£f" 
between the demands of good measurement and the demands of reality. 
The major factors which need to be considered generally in re** 
lation to practical matters are cost, time required for admlnls*” 
tratlon and scoring, ease of administration and scoring, the 
availability of comparable test forms, user acceptability and 
potential usefulness of results. Ilearly all textbooks in ed^- 
cational and psychological measurement deal with these topics, so 
they will not be pursued here. 



MEAS»*SEMENT IN INSTRUCTIONAL SYSTEMS RESEARCH 



As indicated earlier a rather sharp distinction can be nade 
between Ijwtnictlonal systems evaluation and instructional systems 
research; instructional evaluation pertains to that class of 
activities that leads to the acceptance, modification or rejec- 
t on of a specific instructional system wxl&in a specific context: 
tostructlonal research Involves that clase of activities that 
leads to the extension of generalizable knowledge . Evaluation 
tovolves essentia3.1y three functions: 1) the determination of 
^ether a l^tjicr achieves the level of performance that Is expect- 
ed of him, 2) an analysis of the Instructional system being used 
to determine those aspects of it that are effective and those that 
are not, and 3) an analysis of the appropriateness of the con- 
ceptual framework that underlies the instructional system in 
light of the empirical evidence that derives from analysis 2. In 
this sense instructional systems evaluation is a continuous, re- 
veling process that leads to an increasingly clear statement 
of te^al objectives, theli enabling objectives, and the In- 
structional system that leads to their development. 



Instructional research Involves quite a different set of 
functions and a much broader focus. In contrast to the develoo- 
^nt-testlng-development cycle, in instructional evaluation, with 
xts focusj upon products T research Involves a hypothesls-testing- 

a focus upon facts or prlncioles. A 
parage* or «a ueable product” evolves from STevaluatlon 
activity, and it may or may not be generallzable , ’’words” 
evolve from a research activity and are generallzable . Also, 
in conttMt to the purely descriptive-analytic nature of evaluation, 
the strategy of research is to compare, correlate and predict. 

Its aim is to test such matters as the appropriateness of an 
iMtructicmal system for various kinds of children or the effect 
or the system under various kinds of conditions and with 

various Wnds of teachers. As such, it requires the introduction 
o experimental manipulation and control (research design) 

V® the use of analysis techniques which facilitate the drawing 
Of Inferences from observations (statistical analyses). It also 

measurement focus. Instead of being concerned 
OTly with measures of educational achievejasnt or outcome, as Is 
tne case In evaluation, instructional research of necessity 
focuses upon three classes of measures: 1) the treatment conditions 
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that comprise the instructional system (content » media, methods), 
2) the contingency factors vhich Interact T/ith the treatment 
conditions to make them effective or ineffective (learner charac- 
teristics, teacher characteristics, setting characteristics), 
and 3) the outcome or achievement measures. Since the measurement 
of educational achievement has been dealt with in the preceding 
section of the paper the present section will focus only upon 
treatment and contingency measures. 

The aim of this section of the paper is threefold; 1) to 
descrihe the nature of treatment and contingency measures, 2) 
point up the similarities and differences between these measures 
and achievement measures, and 3) to Introduce the reader to O.R. 
Buros* Mental Measurements Yearbooks as a guide to the published 
measures that are available to the educator and educational re-r 
searcher. Because of time and space limitations these comments 
must be brief. 

The Nature of Treatment Measures 

Treatment measures involve essentially detailed descriptions 
of that idilch comprises an instructional system. This Involves 
three separate components: 1) the content of the system, e.g., 

the firepower of a nuclear submarine or the theme of a Wagneriar 
pera, 2) the media used to transmit the content, e.g., the print- 
ed page, pictures, models, records, television, films, real- 
life observation, and 3) the methods used in dealing with the 
media and its content, e.g., individual study followed by an 
exaieination, lecture, discussion vhich follows an "inquiry” 
model, small group discussion, teaching machines. Historically, 
dlscriptions of these components have been extremely gross— aiich 
like they appear above. Within recent years, however, more de- 
tailed descriptions of these components are beginning to find 
their way into the research literature, and as a consequence the 
nature of the data on the relationship of treatment conditions to 
learner outcomes is starting to cake a different shape (cf Tabs, 
1964, Bellack, 1965). 

The detailed description of content, media and method 
provides a set of measurement problems that are essentially the 
same as those that confront the developer of educational achieve- 
ment measures. As a beginning step he must find or develop a 
framework which lets him know what he Is to measure. If he has 



to develop the framework he is doing essentially the same thing 
that the instructions! systems designer does when he breaks out 
the enabling objectives that are in service to a terminal objec- 
tive. If the focus is upon coiitent, the task is to find a frame- 
work which permits a meaningful ordering of the substantive in- 
formation within a unit of subject matter. Depending upon the 
particular focus of the research this could follow the scheme 
of Gagne* (1964) and study content from the point of view of 
structure i.e.. the hierarchy of discriminations, associations, 
concepts, principles, etc. that are Involved in it, or it 
could be approached from the point of view of a topical analysis , 
i.e. the substantive topics covered. If the focus is upon method, 
the task is to find a framework which permits a meaningful ordering 
of the various aspects of a teacher *s behavior that are designed 
to facilitate the learning process. Detailed analysis schemes of 
this kind are now appearing, but as yet there is some question 
as to their utility (Schalock, 1967} . If the focus is upon media 
the task is to find a framework whi<^ permits the classification 
of various media forms. Once a framework has been developed which 
identifies that which is to be measured (described) the process 
of developing the respective measures can begin. 

* 

By-and-large, the development of treatment measures follows 
the same patterns and involves the same set of operations es 
does the development of outcome measures. Both involve direct 
measurement, and as such attention must be given to the issues of 
relevance, representativeness, fidelity and reliability. Since 
measures of treatment conditions have not as yet been developed, 
the measurement task of the educational area in this respect is 
huge. 

The Nature of Contingency Measures 

It will be recalled that contingency factors are those 
factors that Interact with treatment conditions in bringing about 
educational outcomes. As with treatment conditiems there are 
also three general classes of contingency factors: 1) learner 

characteris tics . e.g., age, intelligence, sex, *'leamlng styles", 
background of experience, 2) teacher characteristics. e*g., back- 
ground of knowledge, background of experience, personality 
orientations. Interests, and 3) setting characteristics, e.g., 
number of children In a class, size of classroom in relation to 
nisaber of children in it, physical facilities, time of day. His- 
torically, a great deal of effort has been directed to the measure- 



ment of these factors^ particularly the characteristics of the 
learner » for these are the factors Which determine whether an In- 
structional system will be effective or Ineffective. 

l^lle many contingency factors Involve direct measurement, 
e.g., age, sex, background of experience, nuoiber of children 
In the classroom, physical facilities, etc., mapy do not. Many 
contingency measures are Indirect measures In that they are 
measures of qualities that are only Inferred (constructs). In- 
telligence, learning style, anxiety. Interest, and aptitude 
are concepts of this kind. One cannot see, hear, touch, feel, 
smell or taste these qualities; they can only be inferred from 
that which Is observed. In this sense, the major difference 
between direct and indirect measurement Is the reliance of 
indirect measurement upon that which Is being observed as an 
indicator of the construct that is being measured and the 
reliance of direct measurement upon that which is being observed 
® descriptor of tl^at which is being measured . Since the 
issues involved in direct measurement have already been dis- 
cussed the remainder of the present section is devoted to in- 
direct measurement. 

With such a basic difference existing between direct and 
Indirect measures one would expect the measurement processes 
Involved to be quite different. This is not the case, the 
steps or operations involved in direct and indirect measurement 
are exactly the same. So are the procedures involved in their 
construction. After identifying that ^ich one wishes to isscsurey 
an appropriate measurement methodology is selected, situations 
to be sampled are identified, items are built and the measure 
is then tried out and evaluated. The issues of relevance, 
representativeness, fidelity and reliability are as crucial in 
indirect measurement as they are in direct measurement. The 
one difference that exists in these two classes of measures is 
the requirement that indirect measures must have some empirical 
evidence that they are in fact measuring that which they are 
supposed to be measuring. This calls for an attribute above 
and beyond those of relevance, representativeness, fidelity 
and reliability, namely, evidence of construct validity. 



Ap with the concepts of relevance, representativeness 
and fidelit)^ construct vcilldlty also relates to the issue of 
whether a measure is measuring that which it is supposed to 
measure (constant error). They differ, however, in the kind of 
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evidence that is {^ennlsslble in their support: the attributes of 

relevance representativeness and fidelity involve essentially non- 

OKUpi«-«» AWOA> I OUOJLJ j w v ^viWMWw \ 

or ''content” validity) whereas the attribute of construct vali- 
dity requires experimentally obtained empirical evidence. This 
requirement makes validity evidence costly and difficult to obtain 
(for it requires a full-scale research program to do so), and as 
a consequence only the better "standardised” measures are likely 
to have *r., Savely will teachers be able to obtain validity 
evidence for the tests which they develop within a class. 



Two kinds of empirical evidence are desirable in support of 
the construct validity of a measure: 1) correlation with another 

measure that is known to be a valid measure of the property under 
consideration (concurrent validity), and 2) the experi^ntal 
verification of hypotheses \vtilch Involve the concept being measured 
and which have used the test being considered in the research that 
has provided that verification (theoretical validity). Either 
or both kinds of evidence provides confidence that the measure 
"is in fact measuring that which it is supposed to be measuring.” 



One further comment about construct validity: just as the 

evaluation of an instructional system permits the clarification and 
testing of the conceptual framework which underlies it (the hier- 
archy of enabling objectives) the pursuit of construct validity 
permits the clarification and testing of the constructs used in a 
discipline. In this sense, obtaining evidence of construct valid- 
ity is as much a conceptual or theory developing activity as it 
is a measurement activity. Uhlle most practicing educators will 
not be involved in the development of Instruments to test concepts 
or theory, they need to be familiar with the idea of construct 
validity for whenever they use indirect measures in their research 
they will have to support them with ^ividence of this kind. 



A Note on Prediction 

/ 

Huch of science is involved with prediction, for once pre- 
diction is possible it enables control. Educators are also con- 
cerned with prediction: who will pass and who will fail? Nho 
is a good college risk and who isn't? T^o is likely to benefit 
from curriculum A and who from curriculum B? To permit prediction 
there must be measurement, for one predicts whatever hm is 
predicting from some set cf measures. These may be as general as 
age or sex or socio-economic status, or as specific as number of 
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questions asked or number of comments made in a class. Of signi- 
ficance to the nresent discussion » however, is the fact that when- 
ever ativ maacurc. is included in a generality used prediction scheme 
there must be evidence of Its predictive validity . Predictive 
validity, like construct validity, requires empirical evidence, 
i.e., that the measure does in fact predict that which it is sup- 
posed to predict. 

A Note on the Selection of Available Measures for Use in Instruc- 
tional Systems Evaluation and Research 

Those who wish to consider using existing, standardized 
measures for purposes of evaluation will find the information in 
0. K. Buros*3 Mental Measurements Yearbooks (cf. 1959, 1965) of 
great value. A comprehensive coverage of all published tests 
in nearly all fields of education is available through this source) 
along with excellent reviews of the specific strengths and weak- 
nesses of each. Reliability and validity data are included in 
the reviews. 

Along with the Yearbooks Euros has published a 479-page 
index to Tests in Print (1961). With this reference source it 
is possible to locate rather easily the yearbook in which a 
particular test has been reviewed. To provide some notion of the 
range of subject areas covered by Euros the classification of 
tests included in the 1959 Yearbook is included as Appendix D. 
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APPeiDIX A 



THE NATURE OF MEASUREMENT IN THE BEHAVIOPAL SCIENCES 



The Concept of Measurement 

/1 950 533) Viaa reported that In a sample of 2h million 

words the term ’’measure” occurred ittore than 400 times and was used in 
40 different ways. It was used -idicate the act of weighing » the 
balance in which the welgihing v t -- , the grams that were used to 
balances and the numeral that eiW' -«aed the result. The term also 
referred to less exact instruments, processes, and units. In fact, 
any Instrument that is used as a basis for comparison, even when that 
comparison Involves the process of estimation or Judgment generally 
carries the term "measure”. Thus, as used popularly, measure not only 
refers to procedures that have precision, but also to acts of objective 
estimation, such as the estimation of beauty or the estimation of an 
Individual’s iu:elllgence. 



A Definition of Measurement 

In its broadest technical sense, measurement is the assignment 
of numerals to objects or events according to rules (Stevens, 1951). 
As measurement is conceived within the physical sciences this is 
rather s straightforward definition: using a device such as a 

ruler or a scale in accordance with relatively simple principles 
one can assisn inches to a table top or ounces to a cup of flour 
without difficulty and with considerable (thou^ never exact) ac- 
curacy. It can be a different matter in the behavioral sciences. 

Suppose that we ask a male judge to stand . '^‘vea feet 
away from an attractive young woman. The judge is asked 
to look at the young woman and then to estimate the degree 
to which she possesses five attributes; niceness, 
strength of character ^ personality, musical ability, and 
intelligence. The estimate is to bt given numerically. 

In the number system a scale of numbers from 1 througjh 
5 is used: 1 indicating a very small amount of the 

characteristic in question and 5 indicating a great deal 
of the characteristic. In other words, the judge, 
by looking at the young woman, is to assess how ’’nice” 
she is, how ’’strong” her character is, and so on, using 
the numbers 1, 2, 3, 4, and 5 to indicate the amount of 
each characteristic she possesses. 

After the judge is finished, pother male judge is asked 
tc repeat the prexess with the same young woaan. The num- 
bers of the second judge are checked against those of the 
first judge. Then both judges similarly judge a number of 
other young women. (Ker linger, 1965, p. 41) 
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'Xhe author goes <m to point out that while this example may seem 
ridiculous as an illustration of measurement, it does in fact meet the 
definition of measurement • The Judges assigned numerals to objects accor* 
ding to rules: the objects, the numerals and the rules for the assign- 
ment of the numerals to the objects were all specified, (the numerals 
were 1, 2, 3, 4, and 5; the objects were the young women) and the rules for 
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structlons to the Judges* ^at makes* tha example seem far fetched as an 
Illustration of measurement is the fact that the properties of the chara- 
cteristics being judged were not at all specified i.e*, \diat is meant by 
"nice”, '‘strong”, "intelligent'*, etc., the rules for assijsgiiag the numerals 
to the nropertiee were not clear* i.e* yikit properties had a value of 
1, 2, 3, etc*, and the conditions under Which the observations were to 
be m^e were not spelled out* Without these adnlmai cwsideratlons 
"reproducebility of observations’* is not likely to be readied* This of 
course is one of the first criteria of measurement that scientists and 
users of measures demand for without reproducibility a measure is of 
little value. To become reproducible, the properties of that which is 
to be observed, the rules by which numerals are to be assigned to those 
properties and the conditions under which the observations are to occur 
must be made explicit and public. Uhfortvnately, a definition of measure- 
ment contains no statement about the quality of the procedures involved 
in it. 



While space will not permit an extended discussion o* the Intxl cades 
involved in specifying properties, rules, context, et^ e essential 
nature of eadi of these processes will he noted. 

Specifying that Which is to be Measured 

Perhaps the moat basic of all concepts in measurement is the simple 
noticn that in order to measure something one must know what it is he 
wishes to measure. 'Jhfortunately, like the definition of measurement, 
tHs also is a deceivingly simple statement. IVo requirements ate in- 
herent in it: 1) sob;<% concent of that which is to be measured exists 
and 2) some notion of the measureable nronertles of the concept exist. 

Put in another way, that which is to observed is dependent upon man's 
ability to conceive of it and, then, c£ his ability to observe it (Lorge, 
1950, p. 536). 

T he development of concents. An essential feature of any science 
Is the dev3lopas^^t, extension and refinement of its concepts, for it 
is t.hrough its ccmcei>ts that it gains its pcwer. Conceptualization is 
a constant, cyclical data-dependent process wherein new data (observations) 
give rise to new concepts and new concepts give rise to new data. The 
cycle doesn't begin at a particular place, nor end; it is ever present 
and 80 long as man inquires it will forever be present. 

The point that la critical here is that concepts "don't Just happen" 
or "don't first exist." They are man made and they are constantly evolving. 
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For ages, man disregarded certain objects or their effects because 
he did not knou of them or their behavior* He did not notice ultra- 
violet radiation, nor the fact that quartz reacted differently from 
glass to ultraviolet light* Nor did the notice electrical currents 
In the brain, or "unconscious’* motivation or that some people can taste 
certain things and others cac*ti Ws nov have these concepts , and they 
have opened an apprecl€d>le store of knowledge* We don’t have them 
In the same form as they were conceived initially however* Atomic 
structure 9 circa 1967, Is not atomic structure circa 1937* Neither Is 
"Intelligence" nor **motivating** nor '‘learning"* Concepts come and 
go, maybe powerful or weak, but aXwa^j they change* Moreover, whatever 
their form, they dictate that which is to be observed and measured* 

The Identlficstlon of the Properties of Concents* Having 
Invented a concept, or having revised one. Is not enouidi In itself 
to permit Its measurement* The properties or characteristics of the 
cortept must also be specified, for it Is these that are In fact 
actually measured, l*e* receive the assignment of numerals to indicate 
quantity* Thus it is not sufficient simply to Invent a concept such 
as intelligence, or achievement, or anxiety, or poverty, or cultural 
deprivation . One musu also specify what it is that constitutes these 
concepts* 



This constitutes one of the most difficult problems in measure- 
ment for as yet there are no rules governing how It is to be done, 
and no way of knowing whether It ever gets done* The procedure followed 
In specifying the properties of a concept has been labeled generally 
as "constitutive definition", (Margenau, 1950) (Torgerson, 19S8), and 
refers to a procedure whereby a concept is sl^ly defined with pro- 
gressively lower-order concepts until one either encounters a set of 
cmacepts whose properties have been defined in other terms or 
reaches a point beyond which defining concepts can't be found* Thus, 
one may begin with a concept such as teacher behavior, break it down 
into its major components, e*g^, careteking, teaching, and routine- 
administration behavior, then break each of these down into their ; 
major components, etc* Ultimately a point is reached beyond which 
lower-order concepts are not applicable, and at that point the pro- 
perties of a concept ^t 111 have been specified so far as available 
knowledge permits* The assumption underlying this process is that 
It is at this level that measurement operations, the assignment 

of numerals to properties by rules, should take place* This whole 
process will be recognized of course as a procedure comparable to the 
hierarchical analysis of educational objectives that Dr* TWelker has 
reviewed in the institute* 

One aspect of measurement that is bothersome to people is hl^- 
ligihtcd in the previous discuaslcm, naitely, the fact that any single 
aaasure can attend to only one or a few of many properties of a 
complex concept* No single measure of a teacher's behavior can ever 
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measure all o£ the properties or dimensions of her behavior; nor 
can a single measure of Intelligence ever measure all of the rich 
and diverse pro^rtles of Kuman Intelligence, or a elngle measure 
of creativity ever measure all of the characteristics of human 
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erally an effort Is made to measure a property or characteristic of 
an object with little or no regard for any of Its other properties 
or characteristics. The length of a table may be estlnated without 
reference to its color, width, height, wood, style, or shape. Psycho- 
metrlsts try to make estimates of the ''Intelligence*' of an adult 
with little or no consideration of his race, personality, or economic 
circumstance. Be this as It may, a single measure can at best give 
only an approximation to a concept, and If the vconcept Is one which Is 
complex, l.e. Involves many properties. It will require many measures 
to yield an adequate approximation of It. 



Specifying the Rules by Which Numerals are i^slgned to that Which Is 
to be Measured. 



AssuBilng that the properties or characteristics of a concept 
have been identified (defined, conceptualized) there Is still the task 
of assigning numerals to these characteristics in such a way as to 
indicate their quantity. This procedure must be governed by "rules" 
or "operatlcne" which, like the properties of the concepts being 
measured, are made explicit and public. Only In this way can a measure 
be "reproducable", and thus admissible as dbjectlve Information. 

In thinking about the assignment of numerals to properties 
It needs to be recognized that a numeral Is a symbol of the form: 

1, 2, 3, ..., or I, II, III .... It has no quantitative meaning 
unless we give It such a meaning. It is simply a symbol of a special 
kind. The term numeral is used because measurement ordinarily uses 
numerals which, after being assigned quantitative meaning, become numbers . 

^ ppmbct is a numeral that has been assigned quantitative meaning. 

As used here the term "aaslgned" refers to the mapping of the 
objects of one set (numerals) t^ the objects of another set. In behavioral 
science research the members of one set are ustially individuals, or 
properties of concepts which relate to Individuals , and the members of 
the other set are usually numerals. 

Rules govern the assignment of the objects of one set (numerals) 
to the objects of another set (persons or properties) , In the 
behavioral sciences a "simple minded" rule might say: "Ass.'^yi the 

numerals 1 through 5 to individuals according to how nice they are« 

If an Individual is not at all nice, let the number 1 be assigned. 

Assign to individuals between these limits numbers between the liMts." 
Another rule might be: "If an individual is male, assign him I* If 

an Individual Is female assign her 0." Both rules of course assume 
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previous definition of concepts and isomorphism with reality." (Kerlinger» 
1965, p. A13-417). 

As with anything else the rules of measurement may be "good" or 
"bad". To the extent that they are good or bad, measurement is likely 
to follow Suit. Many things are relatively easy to measure because 
the rules are easy to draw up and follow. To measure sex, for example, 
is easy since several simple and fairly clear criteria can be used to 
detemsine sex and to tell the investigator when to assign 1 and when 
to assign 0. It is also easy to measure certain other human character- 
la ticus hair color, eye color, height, weight. Unfortunately, most 
human characteristics are much more difficult to measure, mainly 
because it Is difficult to specify clearly the properties of the 
characteristic to be measured and to devise clear rules that govern 
the assignment of numerals to them. Nevertheless, rules of assignment 
must always govern the measurement process, and more and laore attention 
must be directed to the specification of such if the behavioral sciences 
are to advance. 

Specifying the Conditions Under which Measurement is to Occur 

In scientific observations, whether direct or indirect, the conditions 
for observation must be carefully specified in terms of time, place, and 
circumstance. In physics and chemistry, observations at sea level at 25^ 
centigrade may differ markedly from observations of the same thing at Qo 
centigrade and in an airplane 35,000 feet above sea level. In psychology, 
the behavior of an Individual at 2 A.H. at his desk in his own home may 
differ markedly from his behavior at 10 A.M. at his desk in his office, 
or the behavior of a teacher may differ markedly in one subject area as 
compared to another or when working in a reading group with "good" readers 
as compared to "poor". 

To control for this source of variance all measures necessarily 
must specify the conditions under which observations are made. 

Measurement As a Sense Datum 

As terge has pointed out (1950, p. 533-539), empirical measurement 
ultimately depends upon the occurrence of a sense datum and its inter- 
pretation by an observer. Melght, even though measured by machine, is 
ultimately related to the kinesthetic sensation of heavier and lighter. 

The machine— that is, scales— merely allows for a simple and relatively 
objective porception of the effects of x^eight. The geiger counter is 
a machine that enables the observer to perceive a specified class of 
effects by extending the range of human sensation. A test of intelligence 
serves the same purpose by standardizing a set of stimuli and a set 
of rules for applying the stimuli to the assessment of given properties 
of an individual. Other machines facilitate measurement by the control 
uf aystematic, chance, or erratic conditions, or by magnifying effects. 



In this sense machines and tests simply allot? more precise determin- 
ations of a particular* class of effects. In the absence of instruments , 
fot ‘extension of the senses, oif for the control of conditions, human" 
observations are liable to error. InstrSntents ere a means for approxi- 
mating more closelv the nrooertv under observation. 
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Unfortiinately. there is a limit to the closeness of approxi- 
mation In the measurement of a property by an observer or by an instru- 
ment. In calorimetry It Is well known that the temperature of the 
measuring device affects the ts^^perature of the materlcil under obser- 
vation, and In this respect observation must be corrected by calculation 
for the Influence of the instrument. In Intelligence testing an 
examiner can affect the score of the candidate by providing encourage- 
ment or its lack.. In the, physical sciences a great deal of attention 
is devoted to the reduction of the Interaction of the instrument with 
the characteristic under observation but in the behavioral sciences 
this is rarely considered. Ultimately, however, the behavioral sciences 
also must learn to control for measurement effect. Until that time 
all one can do is to be aware that it exists and do all that is possible 
to reduce it. 

Characteristics of Measurement in the Behavioral Sciences 

In the previous section attention was given the fact that in 
any measurement the act of measuring is directed to the properties 
of objects or concepts rather than the object or concept itself. 

While this is true, it does not mean that the properties theiKselves 
are actually measured, for the properties of an object or concept are 
themselves concepts. What actually gets measured in the act of measuring 
are indicants of the properties of objects, this is the cac*; in all 
measurement, whether in the physical or the behavioral sciences. 

The problem of Inference 

Objects, processes, etc., or concepts of objects or processes, can 
be thought of as falling along a continuum of concreteness-abstractness. 

If one conceives of that which is to be measured in this way it becomes 
clear that soiae of the properties of objects, e.g., width, height, 
wei^t and hardness fall on the "concrete" end of the continuum and are 
therefore amenable to rather "direct" observation. This is pot the case 
for such concepts as intelligence, morale, anxiety, hostility, or creat- 
ivity. These are concepts that exist only by Inference; they are assumed 
or Inferred to exist because of certain obsorved regiilaritles in' behavior. 
As one moves away from the measurement of concrete, ‘^'naturSl" objects, that 
is, objects that are available to direct observation, and into the measure- 
ment of the properties of "inferred" objects or concepts, reliance 
upon indicants becomes heavier and heavier. As this happens uneasiness 
about measurement increases. As soon as relatively simple physical 
properties are left behind for more complex and elusive properties. 



direct observation of properties Is Impossible. Hostility cannot be 
observed directly, nor can morale, anxiety, intelligence, creativeness, 
talent, etc. Such concepts require Indirect measurement in that their 
properties or characteristics must always be inferred from observation 
of presumed Indicants of those properties. 

Hhen using Indirect measures Indicants are taken as reflect** 
ions of an underlying property or characteristic. They are seen as 
’’something which points to something else.” If a boy continually strikes 
other boys, we may say that his behavior is an Indicant of underlying 
hostility c If someone's hands sweat excessively, we may say that be is 
anxious. If a child answers a certain number of items in an intelligence 
test correctly, we say he has a certain level of intelligence. In 
each of these cases, some identifiable behavior is assumed to be an 
indicant of an underlying property. 

It is rather understandable that people view measurement as being 
rather shaky when it involves making inferences from observed behavior 
rather than directly observing properties like skin color, size or sex. 

To measure a child's cooperativeness, dependency, or imaginativeness 
is very different from measuring his height, weight, or wristbone 
development. The fundamental process of measurement is the same but 
the rules are much more difficult to prescribe. Inference dominates, 
and this creates one of the more vexing problems of psychological and 
educational measurement. 

Operational Definitions 

The use of operational definitions represent an attempt to over- 
come the problem of inference in the measurement of abstract concepts 
or constructs. 2 In essence, an operational definition is a definition 
that assigns meaning to a construct in terms of the operations that 
are used in measuring the indicants of that construct . Thus, if only 
one measure of a construct is used, intelligence may be defined as 
that which is measured by test X, or anxiety defined as the number of 
facial ties emitted over a given period of time and under specified 
conditions. As indicated earlier, a single measure rarely taps all of 
the properties of an object or a construct, and, as a consequence, 
more than one measure per construct usually should be obtained. 

While operational definitions are indispensible Ingredients in 
all scientific measurements they are dependent upon the conceptual 



^Constructs are concepts which have the added meaning of 
being deliberately or consciously created for a special scientific 
purpose. Intelligence is a case in point. 





activity that breaks out the properties of the concepts or constructs 
that are to be roeaeured. Either activity by Itself is relatively mean- 
ingless. In combination, however, they constitute the essence of the 
scientific enterprise, namely, the shuttling back and forth between 
the level of construct -hypothesis-theory building and systematic ob- 
servation. 

the Problem of Error Variance 

Even though a measuring instrument reflects a careful conceptual 
errort, sensitive operational definitions, and a detailed statement 
of the rules for assigning numerals to properties, error V7ill still 
enter the measures taken with it. Two people using the same yardstick 
to measure the same table will come up i^th different results. A 
chemist uses chemical balances so sensitive that they must be kept 
In another room to guard against the Influence of body heat or of air 
currents set in motion by the chemist’s movements; nevertheless, he 
will weigh material several tines and still settle for an average as 
ibe ’’true” wel^t. The control and/or elimination of error in measure- 
ment is one of the most critical tasks of the scientist. 

With physical measurements subject to error, it should not be 
surprising that behavioral science measurements are stl’ < more so. 

For example, attempting to measure intelllgcmc^ with paper- sad=pe?»cil 
tests would seem to Invite all kinds of error* And it does^ hut measures 
of Intelligence also reflect to some degree the trait being measured; 
they are not totally inaccurate. If a persen scores high on an arith- 
metic test, for example. It is reasonable to assume, despite the like- 
lihood of error, that he Is pretty good at arithmetic. 

Essentially, this is the basic assumption of measurement in the 
behavioral sciences; any measure contains an element of error and an 
element of truth. Mathematically the assumption is that any obtained 
measure X is the algebraic sum of a true measure t and a measurement 
error e, or 

X • t + e 

Unfortunately, the statement is not entirely satisfactory. Log- 
ically, a ’’trise” score Is one that Is not contaminated by any kind of 
error, but two kinds of error always exist In eve ry measure; constant 
error and random error . By definition, «. constant error is one that 
appears consistently in repeated measurement; random error Is error 
that influences different measurements to different degrees. Now, if 
®*^ly random errors needed to be considered, the mean of repeated measure- 
ment would tend to give a good approximation of a "true” score because 
such errors are not correlated. But since constant error also occurs 
with each measurement taken ^ it influences the mean as much as it in- 
fluences each individual measure. With a distinction between systematic, 
repeatable errors and those that vary randomly, it is possible to 



rephrase the basic equation of testing as 

X » 8 + e 

with 8 (systematic measure) representing a composite of a true 
measure and any constant error* In this revision of the equation, 
e xrepresents only the residual sarror which Is rand. - and unpredictable, 
me/: M. oo\ 

la considering constant error, it needs to be recognized that 
two kinds of constant error exist. One can be attributed to the 
measuring instrument or to a measurement situation. In this case 
ail objects or people measured are affected in the same way; that is, 
each measure in a total distribution is In error to the same degree. 

Other constant errors can be attributed to characteristics of an 
individual being measured. Repeated measure of a trait on one person 
may be consistently wrong for him to the same degree and in the same 
direction, for example, a person with chronic aiixiety may consistently 
score 20 points too low on a given test every time he takes it. This 
is a constant error for him, but it does not Influence other scores in 
the distribution. 

This distinction between error that is constant within a whole 
array of nseasures and error that is constant for a specific individual 
is important, of course, only in relation to a whole set of me^ures. 

In the first case, the error is not likely to make a major difference 
in the interpretation of ones data since it does not upset relative 
positions within the distribution o In the second case the error has 
considerable practical eigniflcaacs because it does produce relative 
changes in the distribution. Statistically, this means that the first 
kind of error produces no variance, while the second kind of error 
has its own distribution and offers a variance of true scores. 

Categorization and Scale Placement vs. t’teasurement 

Ordinarily when one uses the term measurement he has as a referent 
something as manageable and stable as inches or yards or pounds or 
cents; something that can be added, subtracted, divided and multiplied. 
Unfortunately, measures in the behavioral sciences do not contain the 
characteristics which permit all of these operations, and because they 
don’t there is serious debate about what one can in fact do (statist- 
ically and otherwise) with them. The purpose of the present discussion 
is to review briefly the concept of levels of measurement, identify 
the level of measurement reflected in most behavior^ science measures, 
and then speak briefly to the implications of this for statistical analyses. 
In the discussion which follows I have drawn heavily from the work of 
Cooiid>8 (1953, pp 472-485) and Ker linger (1966, pp 419-428). 



Postttlatga of Measureiaeat « Before discussing levels of aea~ 
surenent per sc $ sooite idea of three of the basic postulates of 
oeesurenent is needed. This nay sound a bit forbidding to those 
who are not attuned to nathenatics, but" In fact it is as straight-- 
forward ^ it is basic. A postulate is an aseunption that is an 
essential prerequisite to carrying out some operation or line of 
thinking. In this case» it is an assucq^tion about the relations 
between objects being aeasured. The three postulate may be written 
as follows: 

1. either (a + b) or (a ^ b)» but not both 

2. If .a • b) and (b • c) then (a • c) 

3. If (a > b) and (b > c) then (a > c) 

The first postulate says: *'a is either equal to b or not equal 

to b» but not both.” This postulate is necessary for elassif teat ion. 

One must be able to assert either that one object is the sane in a 
charactcriotic as another or that it is not the sane. In neasurenent 
”the sane" does not necessarily mean complete identity: it can nean 
"sufficiently the same to be classed as members of the sane set." 

Saying that two boys are the "sane" in maleness is in one sense accurate 
(they are both males) but it is likely that one boy may actually bc; 
more nasciiline than the other. This is a criterion matter. To be 
able to say that "the ^o are the sane," one must only meet the crlter-- 
Ion or the set of criteria that have been established as a measure of 
saneness. All criteria hove one requirement, however: they muet be 

sufficiently unambiguous to nake classification possible, that is, to 
satisfy the condition the postulate states. 

The second postulate says "If a equals b, and b equals c, then 
a equals c." Put in other terms, if one member of a universe is the 
sane as another member, and this second member is the same as a third 
then the first member is the same as the third member. This 
postulate enables a researcher to establish the equality of set members 
on a particular charactevistlc. 

The third postulate is of more immeaiate and practical Importance 
for most measurement considerations in the b^avloral sciences. It 
says, "If a is greater than b, and b Is greater than c, then a is 
greater than c." This Is the so-called transitivity postulate. Other 
syitibolo or words can be substituted for "greater than" (">") and "less 
than" ("<") e.g., "is at a greater distance than," "Is stronger than',' 
"pjrecedes," "dominates," and so on» Most measurement in psychology and 
education depends on this postulate, for a goal of most measurement 
to be able to assert ordinal or rank-order statements like "a has 
more of a property than b; b hm more of the property than c; therefore 
a has more of the property than c." 







The preceding ctatexneats may seem obvious , and in physical measure- 
ments the postulate is often satisfied: if stick a Is long?; t|jan 

stick b, and stick b is longer than stick c, then stick a 
longer than stick c. If student a has more items right on a test th^ 
student b, and student b has more right than student c, student a must 
T- - r . But take the relation doinlnance. a 

may dominate b and b may dominate c, but it *s ^ossi^Je th^ a do 
not dominate c. A wife may dominate her husband* and the husband may 
dominate their child* but the child may doadnete his mpther. Xf an 
Investigator Is studying dominance relatione wng ® ^ 

simply assume that the postulate Is correct. He must demonstrate that 

It is correct. 

Most authors assign four levels of measurement: nominal, ordinal, 

Interval and ratio, though (1953) 



Levels oZ Measurement 

1. llte nomlpal scale . Meesureneiiit In tts simplest form consists 
of sabstltating symSols or names for real objects. 
consists only In this mapping of objects Into symbols, the symb 
constitute a nominal scale. Thaa a system nhich classifies 
Into families or the symptoms of patients 

tlons represents a nominal scale. To tie the of leyeW 

oeaaatement to the postulates of measurement, the 

to*' or "not equal to" must hold betiwen objects on a nominal scale. 

This means that any pair of objects must clearly belong to ^ 
class or not belong to the same pleas. In addition, 
equality must be aymnetric and transitive. By sym^tpr is meant that 
if the relation holds between a and b, it also holds between * ® » 

symbolically. If a - b, then b - a. By transitivity Is meant that 

if a “ b and b ■ c, then a c# 

This level of measurement is so primitive that it la not always 
recognized as measurement, but it is a necessary condition tor all 
higher levels of measurement# 

2. Ordinal Scale. Sometimes the objects In one class of a nominal 
are more than just different from the members of another ol“S" 

kind of a relationship to them. ^® ®“* 
ahip is that the members bf the one class ate Mte f 
the members of the other class, and It Is nmanlngful *^® * . 

relation "greater than" (>) holds between the members of °“® c^®*® 
and the members of the other in relation to some P*°P®'*y* 
relationship holds for all members of the two classes the result la 

an ordinal scale. 

f 

3. The T nt..rval and the Ratio Scale. In the two scales discussed 
beretoforS^^^Wnal, and ordlnal-~the elements of the system were 

of objects, and the relationships were relationships of 
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equality and greater than. Nothing was said about a concept of 
distance between classes. Thus, although a may have been observed 
to be greater than b, and b greater than c, nothing has been said 
about a being greater than b by a larger amount than b was greater 
than c. In interval and ratio scales the concept of distance between 
classes enters the picture, and with it significant increase in the 
power of measureioent . The increase in power derives from the fact 
that once information is available on how large the intervals are in 
the property of the object being measured it then becomes possible 
to apply arithmetic operations to them. Interval scales provide 
data that are subject to addition and subtraction, (though it should 
be noted that with t his level of data it is not quantities or ^^^*ounts 
that are added or subtracted, but intervals or distances) . It 
is only when one is able to use ratio scaler, however, that all of 
the arithmetic functions can be applied, that is, multiplication and 
division as well as addition and subtraction. This is made possible 
by the fact that a ratio scale, in addition to possessing the char- 
acteristics of nominal, ordinal, and interval scales, has an absolute 
zero that has empirical meaning. If a measurement is zero on a ratio 
scale, there is a basis for saying that some measured object has 
none of the property being measured. Numbers on the scale indicate 
the actual amounts of the property being measured; if a ratio scale 
of achievement existed, for example, it would be possible to say 
that a pupil with a scale score of 8 had an achievement twice as 
great as a pupil with a scale score of 4. For this reason ratio 
measurement is the ideal of all scientists. 

Unfortunately, measures in the behavioral sciences are far from 
reaching the ideal of ratio measurement, and there is no reason to 
believe that they even will. Most measurement is of the nominal and 
ordinal variety, though many psychological and educational measures 
are treated as if they approximated interval measurement. The impli- 
cations of this are two fold! 1) behavioral science measures of 
necessity each in precision, and 2) the statistical analyses that 
can be applied most to behavioral science data are limited. Dr. Bealrd 
speaks to this point at length in the chapters upon analysis « 
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APPENDIX C 



MULTIPLE CRITERION MEASURES FOR EVALUATION OF EDUCATIONAL OUTCOhLS* 



Indicators of Status or Change in Cognitive and Affective Behaviors^ 
of Student in Terms of Standardized Measures and Scales (Obtrusive 
Measures) 

Standardized achievement and ability tests* 

Standardized self inventories designed to yield measures of adjust- 
ment, appreciations, attitudes, interests, and temperament; 

Standardized rating scales and check lists for judging the quality 
of products in visual arts, crafts, shop activities, penmanship, 
creative writing, exliibits for competive events, cooking, typing, 
letter writing, fashion design, and other activities; 

Standardized tests of psychomotor skills and physical fitness. 

• Indicators of Status of Change in Cognitive Mid AffecUj^ , Behaviors, 
of Students by Informal or Semiformal Teachermade Instruments, or 
Devices (Obtrusive Measures) 

Incomplete sentence technique: categorization of types of responses, 

enumeration of their frequencies, or rating of their psychological 
appropriateness relative to specific criteria. 

Interviews! frequencies and measurable levels of responses to formal 
and informal questions raised in a face-to-face interogatlon . 

Peer nominations: frequencies of selection or of assignment to leader- 

ship roles for which the sociogram technique may be particularly 
suitable . 

Queotionnaires: frequencies of responses to items in an objective 

format and numbers of responses to categorized dimensions developed 
from the content analysis of responses to open-ended questions. 



* From a paper by Metfessel, N.S. and Michael, W.J. A paradigm 
involving multiple criterion measures for the evaluation of the 
effectiveness of school programs. Paper read at the 1967 AERA 
conference in New York . (Mimeographed) . 
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Self-concept perceptions: measures of current status and indices 

of congruence between real self and Ideal self— often determined 
from tise of the semantic differential or Q-sort techniques. 

Self-evaluation measures: student’s own reports on his perceived 

or desired level of achievement, on his perceptions of his personal 

and social adjustment, and on bis future academic and vocational 
plans. 



Teacher-devised projective devices such as casting characters in 
the class play, role playing, and picture interpretation based 
on an Informal scoring model that usually embodies the determin- 
ation of frequencies of the occurrence of specific behaviors, or 
ratings of their Intensity or quality. 

Teacher— made achievement tests (objective and essay), the scores 
on which allov? inferences regarding the extent to which specific 
instructional objectives have been attained. 

Teacher-made rating scales and check lists for observation of 
classroom behaviors: performance levels of speech, music, and 

art; manifestation of creative endeavors, personal and social 
adjustment, physical well being. 

III. Indicators of Status or Change in Student Behaviors Other than 
Those ^^ured ^ Tests, Inventories, and Observat ion Scales in 

^ the Task of Evaluating Objective s of School Programs 
(Nonob true ive Measures) ® 

Absences: full-day, half-day, part day and other selective indices 

pertaining to frequency and duration of lack of attendance. 

Anecaotal records: critical Incidents noted including frequencies 

of behaviors judged to be highly undesirable or highly deserving 
of commendation. 

Appointments: frequencies with which they are kept or broken. 

Articles and stories: numbers and types published in school news- 

papers, magazines, journals, or proceedings of student organizations. 

Assignments: numbers and tjrpes completed with seme sort of quality 

rating or mark attached. 

Attendance: frequency and duration when attendance is required 

or considered optional (as in club meetings, special events, or 
off-campus activities) . 



o 
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Autobiographical data: behaviors reported that could be classified 

and subsequently assigned judgmental values concerning their appro- 
priateness relative to specific objectives concerned with human 
development . 

Awards, citations, honors, and related indicators of distinctive 
or creative performance: frequency of occurrence or judgments 

of merit in terms of scaled values. 

Books: numbers checked out of library, numbers renewed, numbers 
reported read when reading is required or when voluntary. 

Case histories: critical incidents and other passages reflecting 

quantifiable categories of behavior. 

Changes in program or in teacher as requested by student: frequency 

of occurrence. 

Choices expressed or carried out: vocational, avocational, and 

educational (especially in relation to their judged appropriateness 
to known physical, intellectual, emotional, social eesthetic, 
interest, and other factors). 

Citations: commendatory in both formal and informal media of 

communication such as in the newspaper, television, schor assembly, 
classroom, bulletin board, or elsewhere (see Awards). 

"Contacts”: frequency or duration of direct or Indirect communi- 

cations between persons observed and others. 

Disciplinary actions taken: frequency and type. 

Dropouts: numbers of students leaving school before completion 

of program of studies. 

Elected positions: numbers and types held in class, student 

body, or out-of-school social groups. 

r t^racurricular acvtlvities: frequency or duration of partici- 

pation in observable behaviors amenable to classification such 
as taking part in athletic events, charity drives, cultural 
activities, and numerous servlce-rel-ited avocational endeavors. 

Grade placement: the success or lack of success in being promoted 

or retained; number of times accelerated or skipped. 
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Grade point average: Including numbers of recommended units 

of course work in academic as well as in non-college preparatory 
programs . 

Grouping: frequency and/or duration of moves from one instructional 

group to another within a given class grade. 

Homework assignments: punctuality of completion, quantifiable 

judgments of quality such as class marks. 

Leisure activities: nuinbers and types of 5 times spent in: awards 

and prises received in participation. 

Library card: possessed or not possessed^ renewed or not renewed. 

Load: numbers of units or courses carried by students. 

Peer group participation: frequency and duration of activity in 

what are judged to be socially acceptable and socially undesirable 
behaviors , 

Performance: awards, citations received; extra credit assignments 

and associated points earned; numbers of books or other learning 
materials taken out of the library; products exhibited at competitive 
events. 

Recommendations: numbers of and judged levels of favorableness. 

Recidivism by students: incidents (presence or absence or frequency 

of occurrence) of a given student *s returning to a probationary 
status, to a detention facility, or to observable behavior patterns 
judged to be socially undesirable (intoxicated state, dope addic- 
tion, hostile acts including arrests, sexual deviation). 

Referrals: by teacher to counselor, psychologist, or administrator 

for disciplinary action, for special aid in overcoming learning 
difficulties, for behavior disorders, for health defects or for 
part-time employment activities. 

Referrals: by student himself (presence, absence, or frequency). 

Service points: numbers earned. 
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Skills; demonstration of new or increased competencies such as 
those found in physical education, crafts, homemaking, and the 
arts that are not measured in a highly valid fashion by avail- 
able tests and scales. 



Transiency; incidents of. 
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APPENDIX D 



I 



CLASSIFICATION OF TESTS IN T1?E 
FIFTH 14ENTAL MEASUREMENTS YEARBOOK (1959) 



. 


Paee 


ACHIEVaiENT BATTERIES 


~1 


CHARACTER AND PERSONALITY 


86 


Nonprojective 


86 


Projective 


212 


ENGLISH 


324 


Composition 


356 


Literature 


363 


Speech 


367 


Spelling 


368 


Vocabulary 


373 


FINE ARTS 


376 


Art 


376 


Music 


377 


FOREIGN LANGUAGES 


388 


English 


388 


French 


398 


German 


408 


Greek 


411 


Hebrew 


412 


Italian 


412 


Latin 


412 


Spanish 


413 


INTELLIGENCE 


415 


Group 


415 


Individual 


535 


MATHEMATICS 


561 


Algebra 


575 


Arithmetic 


582 


Geometry 


611 


Trigonometry 


614 


MISCELLANEOUS 


615 


Business Education 


615 



er|c 



MISCELLANEOUS Page 

Computational and Scoring Devices 628 

Education 628 

Etiquette 6^*1 

Handwriting 641 

Health 641 

Home Economics 648 

Industrial Arts 650 

Listening Comprehension 650 

Miscellaneous 655 

Philosophy 655 

Psychology 655 

Record and Report Forms 658 

Religious Education 659 

Safety Education 661 

Socloeconcmlc Status 661 

Testing Programs 663 

MULTI-APTITUDE BATTERIES 667 

READING 721 

Miscellaneous 756 

Oral 767 

Readijiess 772 

Special Melds 780 

Speed 780 

Study Sk:.lls 781 

SCIENCE 799 

Biology 806 

Che!ai.stry 811 

Geology 823 

Physics 824 

SENSORY-MOTOR 831 

Hearing 831 

Motor 832 

Vision 834 

SOCIAL STUDIES 841 

Economics . 850 

Geography 850 

History 850 

Political Science 858 

Sociology 870 
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VOCATIC»?S 871 

Clerical 871 

Interests 879 

Manual Dexterity 901 

necnanicajL Aoxxicy v\jh 

Miscellaneous 920 

Specific Vocations 932 
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APPENDIX E 



CONSTRUCTION OF PROTOTYPE ITEMS 



Eva Baker 



Copyiright 1967 



Southwest Regional Laboratory for Education.' .1 Research and Development 

11300 La Cienega Boulevard 
Inglewood, California 



CONSTRUCTION OF PROTOTYPE ITEMS 



After objectives have been phrased in terms of thf' learner's behavioral 
response to stated stimulus or presentation conditions, even further clari- 
fication for the teacher or producer of instructional materials is requisite. 
The next essential step Involves the production of a prototype item de- 
signed to measure the achievement of the objective. Provision of such a 
prototype performs a variety of functions. First, it may clarify any 
ambiguities In the stimulus or presentation conditions stated in the 
objective. Next it provides a model for additional items for the complete 
posttest, so that the test constructor has a definite guide and can gener- 
ate items with more efficiency. 

"Prototype” means that the item must be a specific example of the 
general type of behavior described by the objective. This in turn implies 
that a particular objective must describe classes of stimuli and responses 
clearly enough that members (l.e., items) of the class can be designed from 
this more general description. 

The way to tell if this is in fact the case Is to inspect the objective 
and item on three particular dimensions: learner response, test directions, 

and presentation conditions . 

The factor of prime importance is the learner response. In many 
instances there is little difficulty incurred or subtlety required in 
making the judgment on the basis of response . If an objective calls for 
the selection of answers by circling correct definitions, the item must 
provide an opportunity for the learner to circle alternatives. Suppose 
one objective calls for the learner to use a particular strategy when 
solving in writing problems of a given type. If the test item only pro- 
vides a place for the learner to record the solution to the problem but 
no place for him to indicate the strategy that he used, then the item 
would not be a prototype of the behavior in the objective. Because the 
objective required the use of a strategy as well as a solution, some way 
of recording the process used to solve the problem must also ba supplied. 

Here is a specific example of the necessity for objective-prototype 
correspondence. Does the item below satisfy the response called for by 
the objective? Circle yes or no below. 

Objective : The learner will describe in an essay the way 

to make a cake. 

TQSt Itero ; Directions ; Circle the best answer. 

The way to make a cake is to 
1. beat eggs separately and fold in 
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2. combine dry ingredients before adding liquid 

3. use a lalx 

Yes No 

* * * 

Clearly the response called for in the Item does not conform to the 
specifications in the objective. An appropriate item might be: 

IN A 500-WOED ESSAY, DESCRIBE THE STEP BY STEP PROCEDURE 
NECESSARY TO MAKE A GERMAN CHOCOLATE CAKE. 

Here, the behavior called for is the same described in the objective. The 
actual response stipulated in the objective must be required by the item. 

The second dimension, importance of test directions, can be well 
illustrated by considering this next objective. 

Objective : Given three sets of words and a stimuliis word, 

the student will underline words with the same 
initial sound as the stimulus word. 

Item : Directions : Underline all words which sou*id like 

the first word. 

BOY: BIRD TOY BAT 

In this case, the directions are ambiguous and the learner could reasonably 
underline all three words if he considered rhyming sounds as well. In this 
case, the underlining looks like the appropriate response to the objective. 

The item might have appeared to be satisfactory had the directions for the 
learner not been included, but the same exact behavioral response (under- 
lining words) would reflect a completely different process than that specif- 
ically described by the objective. Correct directions might be as follovjs: 

Directions : Underline all words which have the 

same starting sound as the first word 

A third focus in matching item to objective is observing If the 
presentation conditions specified in the objectives are actually those 
exemplified by the item. If the objective says the learner is to be given 
special Information to assist his performance, the item should oe checked 
to see if such necessities have been provided. 

For example, one’s objective may be that: 

THE LEARNER WILL SELECT THE CORRECT ESTIMATE OF TIME, 

GIVEN THE SPEED AND DISTANCE OF A TRAIN ON A TRIP. 

The item should be checked to see if necessary information is given. 
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For the next objective what presentation conditions must be met? 
Write your answer below. 



THE LEARNER WILL DESCRIBE IN WRITING AREAS OF POPULATION 



nrrwoTnw AT2»m?n TMonwnrpTMr* a iuta-d f\r? oirtBrinn Atimr m ? a cltxjvki 

wuxiUJkAjk cu.' XUM. A .'VIA. vrji: xvra. vxjuvm. A& vrx n vax.xix. 



REGION. 



* * * 

The map that reflects the physical features of the designated area must 
be provided. Prototype items should be checked, then, in three ways, all 
of which are designed to determine if the item is in fact an actual member 
of the class of stimulus ••response combination described by the objective. 

List below the three dimensions one should consider when verifying 
objective-prototype item correspondence. 

1 . 

2 . 

3. 



* * * 

The three dimensions a a: 

1. The learner's actual response to the item should be 
compared to the response called for in the objective. 

2. The directions given the learner to answer the item 
should reflect the Intention of the objective. 

3. The item should exhibit or provide for the presentation 
conditions given in the objective. 

The use of this miniature system of verification should promote high 
objective-prototype correspondence and thus allow the insfn.ictional 
planner to proceed with more confidence. 

For each of the following pairs of objectives and prototype items, 
verify that the item calls for behavior appropriate to the objective by 
inspecting the learner response desired, the test directions, and the 
presentation conditions. Circle "yes” below if the objective and item 
correspond in these three respects. Circle ”no” if they do not correspond 
and check those dimensions which do not correspond. 
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O blectlve t THE LEARNER WILL LABEL A MAP WITH APPROPRIATE 
NATURAL RESOURCES FOR LATIN AMERICAN NATIONS, 

Items Directions ; On a piece of paper list the 

natural resources of the following countries; 

A, (3iile 

B , Paraguay 

C, Guatemala 

YES NO 

Response 

Directions 



Presentation Conditions 

* * * 

You should have circled iw for this pair and checked response and presenta- 
tion conditions. The item makes no provision for the map upon which the 
students are to place the names thereby lacking presentation conditions. 

The response desired is also at variance with the objective since labeling 
a map requires a knowledge of what resources go with which region rather 
than a simple listing. 

Consider this next pair. Answer yes or no below and check the dis- 
sonant dimensions, if any. 

Objective ; THE LEARNER WILL CIRCLE CORRECT EXAMPLES OF 
PATTERN 4 SENTENCES WHEN GIVEN A SET OF FOUR 
SENTENCES. 

Item ; Directions ; Circle all examples of pattern 

four sentences, 

A. The. girl was pretty. 

B. Quietly the wave broke on the shore. 

C. The airplanes roared. 

D. Children are often playfr»l. 

YES NO 

R esponse 

Directions 



Presentation 



* * it 
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You should have answered "Yes,” because the objective and item do corre- 
spond, in terms of learner response, directions and stimulus conditions. 
Try this next set. Answer below. 

Objective r THE STUDENT WILL CIRCLE DESCRIPTIONS OF 

PROBLEMS VJHEH PRESENTED WITH BOTH PROBLEM 
AND NON-PROBLEM DESCRIPTIONS. 

Item; Directions : Which are problems? Circle the 

letter in front of each problem below. 

A. Susan has lost her purse. 

B. Sam has an ice cream. 

YES NO 

Response 

Directions 



Presentation Conditions 



* * * 

You should have, answered "Yes," because again the item and objective 
match on the three stated criteria. Does the follov/ing item adequately 
reflect the objective? Answer yes or no below and check those dimensions 
which do not correspond. 

Objective: THE LEARNER WILL INFER THE POLAR QUALITY OF 

MAGNETISM AND CORRECTLY PLACE OPPOSITE POLES 
OF BAR. MAGNETS TOGETHER. 

Item: Directions: From the picture provided circle 

the two ends of the magnet which attract 
each other. 



YES NO 

R egponse 

Directions 



Presentation Conditions 



* * * 

You should have circled "No" in response to this objective - item com- 
bination, and checked all 3 dimensions since the presentation conditions 
(pictures rather than magnets), directions, and learner response (circling 
rather than manipulating) differ from what is stated in the objective. 
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Fat the ne»t objective to «et*e e 4«Sn totifttog direc- 

tions which would exenpUfy all necessary stipulations ot the objective. 

Write your item below. 

GIVEN FOUR SIMPLE 2-DIGIT ADDITION PROBLEMS, THE LEARNERS WILL 
WRITE THEIR ANSWERS c 

Directions: - — 




A 



* 



* 



lout item had to include four 2-digit problems. Your directions should 
have sounded something like this: 



Dir<£. .lions: Write the answer to each problem., ^ , 

This would be sufficient if you provided addition (+) signs for 
each of the problems. Otherwise you should have stated something 

like this: 



Directions: Add the numberf In each problem and wirite the sum 

' on your paper. * 



The problems could look like this* 



71 

+15 or like this: 
71 + 35 * 



or like this: 

71 plus 35 is ■■ — * 

Yet any of these would be approppiate to the stated objective. 



The writing of objectives and the subsequent clarity 
oroducinc prototype items reflect a procedure considered mandatory befor 
Ly serllti^ Instractlon is possible. Objectives should be 
of the learners' behavior, to be exhibited under particular presentation 

conditions specified in the objective. 



Items which purportedly measure the attainment of the objectives 
should be particular Instances of the class of stimuli and responses des- 
cribed in the objective. The actual correspondence between objective and 
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item must be verified by using three dimensions: learner response, direc- 

tions and presentation conditions > 

After reading through this paper it is hoped that you will be able 
to use specific objectives on your Instxructlonal planning. Your objectives 
should state the specific learner response to given presentation conditions. 
When confronted with objectives which do not meet this standard > you should 
be able to modify them. Further, you should be able to identify and write 
test items which exemplify the objective’s specifications. Such abilities 
will unquestionably result in a greater productivity and efficiency in your 
Instructional planning. 
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RELATIVE VS. ABSOLUTE CRITERIA 



There are two basic methods of describing a learner’s performance. 

The first, and perhaps most familiar to teachers, is that method by which 
a given learner is compared with other learners. The teacher who does 
this uses a relative criterion. The second is that method by which a 
learner’s performance or achievement 1s compared with an arbitrary, pre- 
speclfled standard. This is the use of an absolute criterion. 

The teacher who ’’grades on the curve” is using a relative criterion. 
He has no A Priori knowledge of how much the best , or the poorest , or 
the mean learner will achieve. He does know that , irrespective of the 
performance of any Individual or of the group as a whole* a pre-speclfled ^ 
number of pupils will receive an A and a pre-speclfled number will ’’fail.” 
pie cutting point for the A’s or the failures is not known in advance. 

It is determined on the basis of the performance of all the pupils in a 
given group. Whenever performance or achievement Is assessed in this 
jpanner, the teacher is using a relative criterion. This is, the criterion 
is how well the student performed relative to the group. 

Host teachers have used percentiles at one time or another. This is 
another example of the use of a relative criterion. Consider a typical 
spelling test. 

Prior to the administration of the test, no one knows \diether or not 
any of the pupils will spell all the words correctly. It is conceivable 
that several pupils will score 100% correct, and it is equally conceivable 
that no one will; when one uses a relative criterion, such as a percentile 
rank, this factor is Irrelevant. The assessment is concerned solely with 
ranking each pupil on the basis of his test score: this rank, in the case 
of percentiles, tells one what percent of all the pupils any one pupil 
surpassed. 

To put it differently, a pupil xdLth a percentile rank of 90 sur- 
passed 90 of the 100 pupils in performance on the spelling test. However, 
there is no possible way of knowing how many words he spelled correctly, 
given only his percentile score or rank. 

1. Why is it Impossible to know a pupil’s actual (or raw) score on a 

spelling test, given only his percentile score? 



1, Percentile score is a relative criterion. It compares 
pupils. It does not tell how he has achieved with 
respect to a possible ’’perfect” score. 

ifk'kitltitlfkit'k'k 
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One cannot distinguish between relative and absolute measures merely 
on the basis of whether performance is reported in terms of a raw score. 
Raw scores may very easily be used in a relative criterion context (and 
very often they are used in this manner). Dr. Dale administers an algebra 
test consisting of 43 items. Last year the highest score was 39; and 28 
vrx.b ucwwccu poa94.ug aim xoxxxltg. ctvciryoxie scorCa 

37,38, or 39 was given an A. This year the high score was 42, the cut- 
off point was 32, and everyone who scored 41 or 42 received an A. Dale 
is concerned with how a given pupil compares with the other pupils in 
hils class as far as performance on the 43-item test is concerned. He 
does not decide in advance how many of the 43 -items a good algebra student 
must answer correctly. 

He does not use an absolute criterion* He does use a norm-referenced 
criterion. 



2. Think of a standardized achievement test, such as the Metropolitan or 
the Stanford. Is it a norm-referenced measure or not? 



2. As ordinarily used in a classroom situation , a standardized 
achievement test is a norm-referenced measure. 

But someone may ask, what about the fact that a sixth-grade teacher 
knows, before the test Is administered, how many points a pupil must score 
in order to achieve a result of 7.0 (Grade Equivalent) on the arithmetic 
reasoning section? The answer is very simple. The test was standardized, 
l.e., norms were established for a large sample of pupils throughout the 
country— and the assumption is made that the pupils in this sixth grade 
class are similar to the pupils in the large sample. Before the test was 
standardized no one knew exactly x^here the cutting point would be for 7.0 
or for any other grade equivalent. If, hox^ever, a teacher decided to pass 
only those who achieved a grade equivalent score of 7.0 or higher, he would 
be using this test in the absolute sense, since a grade equivalent of 7.0 
is equal to a specific raw score on the form being used. A teacher uses 
an absolute criterion when he specifies in advance what the minimum accept- 
able performance is. Furthermore, the performance must be described in 
terms of the objectives, and not in terms of hoi^ the pupil compares i/lth 
other pupils. 

If a teacher wants the pupils to give the simis of the 100 addition 
facts, he uses an absolute crltericn. He keeps teaching the pupils— every 
pupil! —until all the pupils in the class have attained the objective. 

The teacher knows before he administers any test just exactly what a "passing” 
performance will be: 100 out of 100. 

Another teacher may be teaching pupils to capitalize correctly. He 
uses proof-reading type exercises, in which pupils must learn to identify 
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and correct errors In capitalization. Later, he administers a test con* 
tainlng Items similar to the practice exercises. Pupils Who achieve a 
perfect score on the test go on to learn other things, while those who 
do not achieve a perfect score receive additional instruction In. capital- 
ization. This teacher Is tislng an absolute criterion. Similarly, If 
he had decided A Priori that any pupil who achieves a score of 90% correct 
or higher will have attained the objective, he would have been using an 
absolute criterion. However, if he had decided that the 90% of the pupils 
In the class who receive the highest scores will have attained the objective, 
he would have been using a relative criterion. 



3. A teacher prepares a 100-ltem test In addition of common fractions. 

He decides that anyone who attains a score of 80 or more correct will 
have achieved the stated objectivi. Is this a relative or absolute dorm? 



3. This Is an absolute. Each pupil Is compared with an 80% 

or better performance. He is not compared with other pupils. 

4. The teacher uses the test In 3, above. He decides that anyone who 
can solve 10 consecutive problems without an error will have attained 
the objective. One pupil may solve the first ten correctly; he is 
finished. Another may solve nine, miss one, solve seven, miss one, 
and so on, and not solve ten consecutive problems correctly until 
he reaches the 75th example. Another misses the first three items 
and then solves "ten-ln-a-row” correctly. The teacher says, "All 
three of these pupils have attained the objective." Is the teacher 
using an absolute or a relative criterion? 



4. Again, an absolute criterion. Pupils are compared only 
with a pre*speclfled standard; ten consecutive correct 
solutions. The pupils are not compared with each other. 



5. "All who achieve a score of 90% on this te^t may move on to the next 
chapter." Relative or absolute? 



5. Absolute. 
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6, The top 90% on this test will pass. The lowest 10% will have to 

take remedial work on this chapter after school. Relative or absolute? 



6. This is a relative criterion. No one knows how high the 
passing score will be. Pupils, therefore, are being c >m- 
pared V7ith each other, not with a pre-specifled behavior. 

iciticititititit** 

?• “When 90% of the pupils achieve a score of 80% or higher, we will move 
on to the next unit." Relative or absolute? 



7. Absolute, actually, there seems to be a relati^ standard! 
here — 90% of the pupils — but this is misleading. The 
critical element here is ”80% or higher" on the test. 

This is a pre-speclf ied , absolute requirement. The 90% 
of the pupils simply limits, or lowers, the absolute 
standard • The entire class will go to the next unit 
when a pre-speclf ied standard is attained. 

ititicicitifk-k 

If, however, one said "The first 90% of the class to attain a score 
of 80% or higher will pass, while the others will not," one would be mixing 
two standards. I'Jhile the 80% or higher remains absolute, a time element 
is introduced as an additional criterion. Pupils will be compared with 
each other with respect to the time required to attain a score of 80%. No 
one can tell, in advance, how long it vyill take the slowest of the top 90% 
to attain the objective. Nor does it make any difference whether it takes 
a week, a month, or a semester. As long as he is one of the first 90% to 
achieve the specified score, he has "made it." But if the pupil is one of 
the slowest 10%, he does not pass. Thus with two standards you have (1) 
an absolute standard with respect to score on a test, (2) modified by a 
relative standard with respect to time. 
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V - SYSTEMS DEVELOPMENT 



Evaluation Sheet 



We need to know how well the Ideas and issues in this manual are 
conn&unlcated to you# You are the test audience for this material# 
To remove or strengthen the weak spots, to retain or improve the 
strong ones (if any) , an account of your learning experience^^ 
you read these sections is crucial. Use this sheet as you stu^y 
and jot down your reactions. Don’t be concerned with typographical 
errors. We’re concerned with how the message is coming through. 

SECTION 1 TITLE — 



Give topic, 
paragraph or 
sentence, page 



Importance 

Rate 

1 2 3 4 5 
Low - High 



Clarity Suggested 

l^ate Improvements 

1 2 3 4 5 
Low - High 



Over-all Rating of the Section: Use a 5-point Scale 





Important 






Understandable 




1 

Very 

Low 


2 3 

(Circle one) 


4 5 

Very 
High 


1 

Very 

Low 


2 3 4 

(Circle one) 


5 

Very 

High 



PLEASE PUT ANY ADDITIONAL COMMENTS ON BACK 
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SECTION V 



Instructional System Development 
Dale G. Hamreus 



Thus far in the institute you have learned (1) how to state 
behavioral objectives that clearly indicate what terminal or out- 
come performances are expected of the learner following instruc- 
tion; and (2) how to determine the specifications for achieving 
the terminal behaviors in (1) . The next step to be presented in 
this section » is to develop>^the instructional materials and meth- 
ods that are required to comply with (2) in fulfilling (1). This 
instructional materials development produces what is termed the 
instructional system. 

In this Institute a system is defined as any combination of 
persons » objects, or events united by some form of regular inter- 
action or interdependence in the accomplishment of a specified 
outcome . 

Why should we be concerned with all the fuss and bother re- 
quired to develop an instructional system? Is not our teaching 
at the college level good? Frankly, no! at least not good enough. 
For example, there pre other ways to teach than just the lecture 
method. Yet, it can be readily documented that by far the most 
frequently employed instructional methodology at the college level 
is that of lecturing. XJhy is this? At least two reasons readily 
come to mind: (1) lecturing is probably the rr'^st natural methodo- 

logy for professors to employ; particularly since most professors 
have never received any formal teaching training; and (2) contin- 
gent upon the first reason, continuing increases in college under- 
graduate enrollments force the pedegogically untrained to persist 
in the lecture. Yet, the very fact of increasing mambers of col- 
lege students results in reduced time available to the Indi- 
vidual student and argues strongly for the need of more precisely 
organized instruction. The only way to break out of the lecture 
lock step and improve Instruction is to systematically develop in- 
structional systems that are valid for the purposes Intended. This 
requires that all methods, materials and procedures be clearly id- 
entified, systematically developed, and refined until they effec- 
tively and efficiently achieve what they were Intended to achieve. 

Our present knowledge and skill in the "science" of deve?i.op- 
Ing Instructiona? systems is admittedly limited, however, we have 
moved past the initial trial-and-er >r stages and are able to at 



least prescribe a process that works, as Dr. Gabriel Oflesh, Dir- 
ector of the Center for Educational Technology at the Catholic 
university of America, recently said, we have developed learning 
technology to the point **where we can start shooting with a bullet 
instead of a shotgun.” (1967, p. 26) 

In this section of the manual discussion we'll center on the 
important stages procedures required for developing and vali- 
dating an Instr mal system. These stages have emerged from 
developmental xe^.iarch over the past five years at Teaching 
Research and other research centers. The general process Is a 
systems approach and requli f specific identification and speci- 
fication of all elements df the instructional system along with 
syi^tematlc try-out, analysis and modifications until the desired 
outcomes are achieved. The fj.0^7 chart presented in Figure 1 
shows the major stages in the system. 

Notice that stage 1 identifying behavioral objectives, was 
discussed in Sectldp XI of this manual and stages 2, 4, 5, and 6 
were presented in Section III. Stages 3, 8b, and 9b, dealing with 
measurement and test construction, is discussed in Section IV. 
Stages 7, 8, 9, 10, and 11 are specifically concerned with the 
process of building and validating an instructional system and 
are discussed in detail In this section. Stage 7 is concerned 
with the actual translation of specifications Into the instruc- 
tional product and Involves content specialists, behavioral 
scientists and media experts; try-out of the instructional pro- 
duct to determine the effectiveness of the new system in the 
central issue of stage 8; In stage 9, analysis of the feedback 
from the try-out are discussed; stage 10 concerns the use of the 
analysis for making necessary modifications to the new instruc- 
tional system; and recycling of this whole process to achieve con- 
tinued refinement of the product occurs in stage 11. 

The building and validation s*"ages can be considered culmin- 
ating activ5,ties to the total instructional system development; 
therefore in order for the reader to more fully understand the re- 
lationships among all stages In the Systems Design in Figure 1, 
a brief review will first be made of the other stages and their 
importance to the stages in this section. 

Stage (1): The first stage in the development of an instruc- 

tional system Is to specify the behavioral objectives. First, 
test your memory on what you learned in Section II. In writing 
behavioral objectives, each statement should contain descriptions 

of the following elements? (Write in answers) 1) , 

2) , 3).^ , and 4) . (Turn to 

page V-7 to check your answers. 
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( 1 ) 



Identify 

Behavioral 

Objectives 



( 2 ) 



Determine 

Enabling 

Objectives 



(4) 



Identify 
Types of 
Learning 



(5) 



identity 
Events that 
Provide Con- 
ditions of 
Learning 

V ■ 



< 6 ) 



Identify 
Form of 
Instructional 
Events 





Produce 

Instructional 

Product 


(7) 




(8) 


Conduct 
Try-out of 
Product 






V 


Analyze 

Try-out 

Results 


/ 

(9) 




V 


(10) 


Modify 

Product 

4 








Recycle 


(11) 




Construct 

Performance 

Measures 



4 



(8a) 



Administer 

Tests 



< 



(9s) 



Analyze 

Tests 



Bracketed numbers (1) 
represent stages 



Figure 1 



Major Stages in System Design for Developing Validated 
Instructional Systems 



If we are to achieve precise learning outcomes from the in~ 
structlonal system that we develop, we must know very precisely 
what changes in behavior we expect of the learner and have 
a standard against which we can check to determine whether indeed 
he has actually gotten there. This is the place of behavioral 
objectives. Behavioral objectives are the very comer stone up- 
on which the systematic development of an instructional system 
rests; and without them we need go no further. 

Stage (2): The next stage in the development of an instruc- 

tional system is to determine the enabling objectives. Test your 
recall again. In section III you learned that enabling objectives 

consist of , ^ > etc. 

that enable the student to t he specified . 

(Confirm your answers on Page V- 7 ) • 

We must make certain assumptions regarding «hat the learner 
must know if he is to achieve the stated terminal objectives. 

These assximptlons are based on empirical evidence, research, 
theory, and many times a seat-of-the-pants logic. They become the 
blue print specs that guide us in the development of the instruc- 
tional system. 

Stage (3): At this point in the development of an instruc- 

tional system it is necessary to construct performance measures. 
Detailed InstrviCtlon on this concept are presented in Section 
IV. However, a few important points will be made here to stress 
the Important place that performance measurement holds in instruc- 
tional system development. 

Criterion measures are obviously necessary for determining 
whether the learner meets or exceeds the level of performance ex- 
pected of him for each behavioral objective. If this was the only 
use for performance measures there would only be minimal need for 
this stage to be Included in the system design and no need for it 
to be located opposite stage (2), determining enabling objectives. 
It could just as well be tied directly to stage (1), determining 
behavioral objectives. However, two very important additional 
functions are served by performance measures in instructional 
systems development that argue strongly for performance test con- 
struction to be tied to stage (2), determining enabling objectives. 
First, we need a diagnostic means for determining the validity of 
various steps of the Instructional system. By developing perfor- 
mance tests from enabling objectives we produce a kind of grid 
like structure for assessing all parts and points of the Instruc- 
tional system to determine where weaknesses exist in the system. 
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This notion expands the concept of just listing performance tests 
to assess learner behavior, and permits Inclusion of performance 
measurement as a means of assessing all elements of the instruc- 
tional systems 

Second, at our present level of sophistication, one of the 
weakest links In instructional systems development concerns the 
assumptions underlying the determination of enabling objectives. 
For example, as subject-matter properties to be generalized and 
discriminated become more subtle, such as the musical concept of 
early or late Beethoven, the instructional process become more 
complicated. A major problem In teaching such subtle and complex 
concepts Is in the analysis and definition of subject-matter pro- 
perties. Such analysis becomes more difficult ^en sematic con- 
fusion exists and where there Is disagreement among experts. 
Assumptions regarding what elements of knowledge or tdiat schedules 
of reinforcements, are required to enable the learner to attain 
the desired terminal behaviors, for example, are ap^ to be faulty 
and must be tested. Performance measures designed to test assum- 
ptions upon which enabling objectives have been determined im- 
prove the chances of making the steps in the instructional system 
valid. 

Stage <4); The fourth stage in our systems design flow 
chart is to identify the types of learning represented for each 
objective. In Section III you learned that Gagne distinguishes 
eight types of learning. Three of these types of learning were 
considered more Important than the other types for college level 
instruction. Determine for the three examples that follow which 
type of learning is represented. (Write your answers in the 
appropriate space ; then check your answers on pag«. mjJ7 ) • 

a. The high school third baseman fields the baj.1 from 

the hitter, looks at the runner on second base, then 
throws the runner out at first base. 

b. The college student replans his course schedule as a 
result of dropping one class and taking a new one. 

c. The teacher plays a recording of a Mozart symphony. 

The student raises his hand when the music changes 
from a major to a minor mode. 

Once the types of learning represented in each objective 
have been Identified then strategies by which they can be made 
manifest can be handled systematically in developing the new in- 
structional system. This narrows down the possible array of al- 
ternatives from which to choose In developing the product. 
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St;age (5): When the types of learning have been determined, 

then stage five becomes necessary: i.e., to Identify the events 

that provide the conditions of learning to occur for each object- 
ive. Can you recall the first four steps for instruction of 
principle learning presented in Chapter III? Summarize each 
step below and check your answer on page V-7. 

Step 1. 



Step 2. 



Step 3. 



Step 4. 



By identifying the events that permit the conditions of 
learning to take place, additional closure is reached regarding 
the type of strategy to be used in developing various parts of 
the instructional system. 



Answers to questions: 



Stage (1) 



In wlting behavioral objectives, each state 
ment should contain descriptions of the (1) Aud- 
ience, (2) Behavior, (3) Conditions, and (4) 
Degree . 



Stage (2) 



Enabling objectives consist of component 
actions, knowledges, skills, etc* that enable 
the student to attain the specified behavioral 
(or terminal) oblectives. 



Stage (4) 



The types of learning represented in the ex 
amples are: 

a. Principle learning 

b . Problem solving 

c * Concept learning 



Stage (5) r Four steps required for instruction of . 
principle learning: 

Step 1. Inform the learner about the form 
of the performance to be expected 
when learning is complete. 

Step 2. Question the learner in a way that 
requires recall of the previously 
learned concepts making up the 
principle. 

Step 3. Use cues to lead the learner to 

put the principle together in the 
proper order. 

Step 4. Using a question, ask the learner 
to "demonstrate" one or more con- 
Crete instances of the principle. 



V-7 



er|c 



Stage (6) : The sixth stage of the systea design is to id« » 

jWitify the form of the instmctional event * That is» specify vhat 
form (verbal, visual, etc.) is to be used for the various types of 
learning identified to transmit the content to the learner. 

Perhaps this is our weakest link in translating specifications in- 
to products. We do have some research evidence to guide the use 
of audio and visual elements but this evidence is still very thin. 
We must rely mostly on seat-of-the-pants logic and empirical 
evidence of pilot tryouts to determine the "best" form to use; 
ael even then tine and cost prohibit any large scale trial. 

All the stages in the system design thus far discussed are 
essential to actually building the eystem. Methods for gathering 
input to develop the InstructioncUL system and for determining re- 
quired output of the system have been described in detail. Input 
is to be identified in terms of prerequlsitrui, enabling objectives 
and conditions for learning; required output takes the form of 
specified behavioral objectives. In addition, the means of id- 
entifying instructional strategies along with supporting media 
forms are to be described. It can be assumed that the content 
is selected and will be sequences and that special equipment can 
be determined. Criterion measures, both intersiediate and end 
course, are to be prepared. !fow we must produce tlie parts of the 
Instructional system systematically asseid>le them, try then out 
with learners, appraise their effects, analyse the outcomes, make 
necessary modifications and recycle the whole process until we 
achieve the desired outcomes. 

The development of a complex instructional system necessarily 
includes Interactions among the learners, the teacher, and all 
other elements \diich constitute the system. However, if we are 
to be successful in developing such a complex instructional system 
we must give close attention to the validity of the system. In 
the past the failure to test the validity of instructional plann- 
ing has limited the effectiveness of instruction. Only as all 
aspects of an instructional system are systematically tested can 
feedback begin to direct our attention to portions that are weak. 
But equaUy important to testing is an organised instructional 
system that can be replicated, thereby permitting the validation 
process to oc( r. 

The key elements of a validated learning system are the 
following: 

1). The end point is learning not teaching, all too often 
teachers place so much emphasis on the teaching role that they 
tend to lose sight of whether or not learning took place. 
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2) . Outcomes are achieved through a system* The system in- 
cludes all of the materials, equipment, facilities, procedures, 
texts, program schedules, maintenance, and personnel required to 
produce the end results. 

3) • The entire process must be validated, utixy wxieu uhe 
system is validated can we have any confidence that we are achiev- 
ing what we set out to accomplish. Understand that the emphasis 
here is in testing the pedagogy, not the student. 

Actually, the development of an instructional system is more 
akin to an "engineering process" in learning. The Instructional 
specifications or "blueprints" must be translated into actual 
materials objects and viable procedures by an instructional en- 
gixieey or **contractorj" who in turn must tost out, analyze and 
modify the newly developed system until the finished product 
actually produces the type of performance called for in the orig- 
inal statement of objectives (learners exhibit specified 
terminal behaviors they have learned). 



stage (7): The seventh stage In developing the instructional 

system is to produce the instructional product. This becomes the 
first stage of actually creating the Instructional system — syste- 
matically following the instructional blueprints or specifications, 
to give the system substance, form, and order. The process is one 
of translating written statements (specifications) into prototype 
of materials, equipment designates, and instructional sequences. 

The process is analogous to the Boeing structural engineer creat- 
ing the complex 727 system from a set of blueprints and specifi- 
cations. In both cases the degree of system integrity determines 
the extent of functional success— obviously more noticeable if 
lacking in the 727 system but no less important in the instruct- 
ional system. 

In creating the instructional product it is essential to syste- 
matically follow the instructional specifications. Content must 
be written into statements or translated into visual and/or audio 
forms, and arranged into sequences that will presumably accomplish 
desired behavioral changes. Media forms must be determined for 
each aspect of the content; l.e. whether to use printed forms, 
slides, overhead transparencies, audio tapes, 8mm loops, 16mm 
movies, various combinations of the above, etc. Formats for each 
selected media form must be established; l.e., whether printed 
statements are to be hand lettered or typed and enlarged; whether 
to photograph real objects, caricatures, or abstract symbols; 
whether to use black and white or color. Strategies for sequencing 
the content and media and for involving the learner and the 
teacher must be decided upon which will presumably be most ef- 
fective in producing behavior changes and also interesting to the 
learner. Decisions must be made detailing exactly what content 
elements go first and which should come next; transitions be- 
tween elements must be specified; what actions and routines the 
learners should experience must be planned; how the teacher is 
to interact in the learning situation must be determined. 

In vaddltion to the above, the preparation of a teacher’s 
manual or guide is important. If the instructional system is to 
produce outcomes as designed, the teacher using it must be provided 
a detailed tset of instructions explaining the exact purpose of 
the system and how it should be employed. Simple systems, such 
as a self -instructional program^ usually have minimal teacher invol- 
vement during instruction, however, the manner of introduction and 
follow-up have considerable Importance to its over-all effective- 
ness. Ttie more complex the system the more crucial the role of 
the teacher. 

During the product development it is essential to maintain 
close liaison between content specialists, media specialists and 
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behavioral scientists. To the extent that each knows what the 
other is doing greater developmental continuity will result. 

For example, if visuals are specified to represent certain content 
portions, considerable headway can be made if meetings are held 
with a graphics artist, content specialist and learning specialist 
during which suggested sketches can be quickly prepared, and 
jointly viewed and Interpreted. Similarly, if a media specialist 
can advise on media forms, graphics work can be prepared permit- 
ting more efficient transformation into the final product. 

Even with specification given, numerous decisions must be 
made during the development. For example, altihough content has 
been defined in the specifications, exactly what combination of 
words should be used, what language level represented, what format 
for displays, which examples to use, what exercises to Include, 
etc. Questions such as these must be answered in the actual dev- 
elopment . 

It is also not unlikely that many gaps will exist in the 
specifications which must be confronted by the developer. Regard- 
less of the detail in specification preparation, it must be real- 
ized that still lacking is the sophisticated knowledge necessary 
to prescribe, in the enabling objectives, all the conditions, 
strategies, and forms requ:|.red In the instructional system to 
produce optimum outcomes. ■ 

Several guidelines have emerged that assist the product de- 
veloper in making decj-slons during development. The model pre- 
sented In Figure 2 represents the total continuum of experience, 
from abstract to reality « This model is helpful in deciding what 
media to use and when. 
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Figure 2. The Continuum of Experience 

For clarification of Figure 2, each of the sections in the 
continuum model starting from the center and working alternatively 
toward the ends will be explained. First, direct experience with 
things . This section concerns real objects and ranges from the 
object Itself to models representing the object. Objects are 
presented in three dimensional form and Increase in number of 
representational cues from fewer in mock-ups to most in the ob- 
ject itself. The continuum of direct experiences with things is 
shown in figure 3. 
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Figure 3. Direct Experience with Things Continuum 
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Direct experience with people consists of continuum ranging 
from authentic or real situations through pantomimes of situations 
to demonstrations of situations. These experiences are represent- 
ational cues from fewer in demonstrations to most in authentic 
situations. Figure 4 contains the continuum of direct experiences 
with people. 




^ — rThree Dimensional Experience—^ 

Figure 4. Direct Experience with People Continuum 

The next two sections concern objective codification which Is 
defined as: of or having to do with a known cr perceived object 
as distinguished from something existing only in the mind of the 
subject or person thinking, hence, being, or regarded as being, 
independent of the mind; real; actual. Determined by and empha- 
sizing the features and characteristics of the object, or thing 
dealt with rather than the thoughts, feelings, etc. of the artist, 
writer, or speaker; as an objective description, painting, etc. 
hence, without basis of prejudice; detached; Impersonal. Objec- 
tive codification with visual experiences are represented in 
tw) dimensional visual form and range from representational cartoons 
without color having fewer cues to motion pictures , with illusion 
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of 3 D, in color having most cues. Ordered elements representing 
objective codification with visual experiences are shown In 
Figure 5. 




Figii e 5. Objective Codification with visual Experience Continuum 
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The continuum representing objective codification employing 
audio experiences is presented in figure 6. 




Figure 6. Objective Codification with Audio Experience Continuum 

The tail end sections of figure 2 concern subjective codifi- 
cation which is defined as: of, affected by, or produced by the 
mind, or a particular state of mind 5 or resulting from the feelings 
or temperament of the subject, or person thinking, rather than 
the attributes of the object thouglCof : as, a subjectiye judge- 
ment. In Psychology : existing or originating within the observerfe 

mind and hence incapable of being checked externally or varifled 
by other persons. Subjective codification with visual experiences 
range from abbreviated symbols (shorthand, etc.) having fewer cues 
to symbolic cartoons having most cues. Figure 7 contains the con— 
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tinuum of subjective codification with visual experiences 




Figure 7. Subjective Codification with Visual Experience Continuum 

The continuum of subjective codification with audio experi- 
ences is presented in Figure 8. 




Figure 8. Subjective Codification with Audio Experience Continuum 

The following general guidelines derive from the continuum of 
experience model and are helpful in making decisions about the 
form of the instructional system* 

1. If the object or event to be taught is available and cues 
can be perceived, then use direct experience, ^or example, if 
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youngsters are being taught about the post offices and one is 
available in the Imediate area, it is preferable to take the 
youngsters to the building rather than employing symbolic 
representations . 

2. If the object or event to be taught is not available 
and cues are not readily perceived, use pictoral representation* 

For example, underwater phenomena are generally unavailable for 
direct classroom instruction, therefore objectively codify them 
(present them by vistial means). 

3. If the object or event to be taught makes vibrations and 
these are signlfic .at cues and are available, use natural sounds. 
For example, the sharp rattling sounds emitted from the tall of 

a rattlesnake are more effectively taught by demonstrating the 
natural sound. 

4. If the object or event to be taught makes vibrations and 
these are significant cues and are not available, use auditory 
representations. For example, heartbeat sounds sjnaptomatic of 
various maladies are readily available through objective codifi- 
cation (tape recordings). 

5. If the object or event to be taught is psychological in 
nature, that is, no iconic (image) representation is possible, 
use audio and/or visual symbols. For example, since no one has 
ever seen an atom, subjective codification must be employed in 
instruction. 

6. If the object or event to be taught is confusing, complex, 
and/or hidden use symbolic representation. For example, subjec- 
tive codification (visual and audio experiences) must be used to 
transmit musical ideas. 
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Stage (8): The next stage in instructional systems develop- 

ment is to conduct try-outs of the product . This is basically a 
quality control measure built into the developmental process. 

The concern of this stage is to subject the newly developed system 
to try-outs vrlth appropriate target populations to determine 
whether the content, strategy and supporting facilities, equip- 
ment and materials of instruction actually do produce the desired 
changes in behavior. The primary purpose here is to obtain data 
from observations and other evaluations of the system such that 
weaknesses in the system can be identified and changed. 

Although this stage implies end-of -development try-out, 
actually try— out must be ongoing throughout the entire develop- 
ment. If the instructional development process is not Interrupted 
periodically to take account of what has already been completed, 
faults in the specifications or the production already developed 
are apt to be overlooked thereby resulting In considerable waste 
of time and money at a much later time. For example, as the 
content of the Instructional system is developed into specific 
written statements, unless statements are checked for accuracy, 
tried out for language level, judged for interest level, etc. only 
limited confidence can be held that these elements will indeed 
be adequate at completion. 

The term try-out tends to imply involvement only with appro- 
priate type learners to determine if their behavior xd.ll be ad- 
equately changed. This interpretation is much too narrow and 
would be too slow and limited a process to produce all of the de- 
sired feedback. The term target population used above includes all 
possible sources to provide feedback of the system’s effectiveness. 
For example, in addition to feedback from learners, xnritten 
portions of the system should be submitted to an ’'unbiased” dis- 
cipline expert to check accuracy of content. Unbiased is used 
here to mean some one who has not previously been involved in the 
new system development. Feedback from a learning specialist 
should be sought to check Integrity of examples to learning types 
and the efficacy of the events employed to produce intended types 
of learning. TIedia specialists should check fidelity and quality 
of audio and visual elements; a ’’typical" teacher should react 
to the instructional design to Identify awkward and/or Impractical 
routines. The above are representative check areas and others 
should be engaged as the system suggests. To the extant that 
feedback can be obtained from various specialists prior to involve- 
ment xd-th actual learners, the chances of obtaining maximum bene- 
ficial feedback in the try-out with learners is Increased. 
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Extremely important to the try-out is selecting the individ- 
uals to cooperate. Care should be taken that they represent an 
appropriate level of competence or ability. Learners should be 
representative of the audience for whom the system is intended. 
However} attention should be given to locate Individuals willing 
to take the time and ”go to the bother'* of conscientiously giving 
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"honest" responses. Subject matter} learning} media and other 
specialists should be skilled and obviously competent to advise 
in their respective areas. 



The learner must be thoroughly briefed on what behavior is 
expected of him during try-out. The emphasis should be placed on 
his seriously trying to learn from the materials. However » he 
should be prompted to verbalize about the places he finds con- 
fusing} too difficult} already knom> etc. Further, he should be 
encouraged to comment if he finds dull and uninteresting areas. 



Close observation must be maintained during the try out with 
learners to produce the required feedback. l<Ihereas curriculum 
studies and other materials developments such as commercially 
prepared products , tend to take large segments of the completed 
materials into the classroom for testing, the process here advo- 
cated is to try-c»ut and test small segments of the unfinished 
system with a single learner at a time. The learner is Instructed 
to identify any <»lementB of the materials that are confusing or 
difficult to understand} such as words, phrases, pictures, symbols, 
test items, etc. The try-out monitor makes careful record of the 
location of all such identifications along with any other signi- 
ficant occurrances during the try-out; l.e., puzzled expressions, 
evidence of being bored, undue time required to complete various 
sections, etc. Xf the monitor suspects the learner is having 
difficulties that he is not expressing he should interrupt the 
instruction and tactfully question the learner. Care must be 
taken in questioning not to create anxiety in the learner. If the 
subject becomes anxious the value of that try-out is greatly re- 
duced and it would be better to discount the effort and reschedule 
another try-^3ut with a different learner. It is important to 
conduct try-outs with as much instructional realism as possible, 
l.e., present the learner with Intact instruction beyond the de- 
signed teaching routines. 



As large segments of the new instructional system are devel- 
oped, more emphasis is given to learner try-outs and less to spec- 
ialist feedback* The emphasis shifts from individual to small 
group learner try-out. Less close observation to the individual 
is maintained and more reliance is placed on test results. 
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To the extent that tests have been developed for suh-portlons 
of the new system and their validity and reliability are judged 
sufficiently high, such tests become diagnostic tools for deter- 
mining weaknesses in the materials. 

Certain details accompanying try-out activities are important 
not to overlook. Clearance for using try-out subjects with appro- 
priate administrative level authorities prior to try-outs will pre- 
vent possible subsequent misunderstandings and will greatly im- 
prove the chances of obtaining additional future try-out subjects. 
In some Instances providing transportation for subjects will avoid 
time lost and will preserve schedules. Monitors must be trained 

to try-outs in what to observe and how to behave during try- 
outs. In situations involving total classroom try-outs, detail^ 
instructions to cooperating teachers specifying all required as- 
pects of the try-out are essential. 

When all development of the instructional system is completed, 
field testing in context is the necessary final stage of try-out. 

By this time, segmented and small scale try-outs should be com*- 
pleted and necessary revisions or modifications to the system pro- 
cessed. At this point it is essential to try the total instruc- 
tional system in the context of an actual classroom with all its 
normal contingencies (expected or otherwise), to see if the 
expected changes in learner behaviors occur. The system should be 
sufficiently complete in all parts — instructional materials, 
teacher guide, tests — that when handed intact to a teacher, class- 
room instruction can be readily imitated. If, in context, the 
system brings the expected proportion of learners to criterion 
within the allotted amount of time then development has been suc- 
cessfully completed. 

Now consider a brief example in which the instructional 
system is at the earlier stages of development. Presume that 
development is underway on instructional materials designed to 
teach 8th graders to use the Readers* Guide. Specifications call 
for visuals in the form of overhead transparencies designed to 
present concepts of the Readers * Guide referencing system and to 
be accompanied by student worksheets containing exploratory state- 
ments and elements requiring application. Several parts have 
been developed: visuals that replicate specific Readers* Guide 

reference entries and work sheet items keyed to the visuals that 
define various symbols and abbreviations used in the entries and 
explain their relationships. One of the prepared visuals contains 
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the following entry: 



Aeronautics: Mid— air payoff. il Time 93:62+ 

J1 21 *67 

The accompanying Student Worksheet had the f ■allowing exercise 
in conjunction with the visual* 

On the screen is a sample of an entry from the 
Readers* Guide. In the spaces that follow, copy the 
following information: Title of article, author, type 

of illustration, name of magazine, volume, page, month, 
day, and year. 

A Teacher’s Guide was also a part of the system and had ex- 
planatory notes for the teacher. The following note was prepared 
for the visual. 

Using the overlay, discuss (1) author and title, 

(2) illustrations, (3), name of magazine, (A) volume number 
and pages, and (5) date. 

Nox 7 put yourself in a hypothetical try-out situation with a 
single learner. Regard the above visual element, worksheet state- 
ment, and teacher note as the segment you are to try-out. Al- 
though a typical try-out segment would involve more than a single 
item series, such as above, for the sake of working through a 
problem example, assume that the above elements are a larger seg- 
ment of an instructional system. In the spaces below identify 
briefly (1) what details you would consider in arranging the try- 
out and (2) how you would conduct the try-out; also (3) guess at 
what effects you might observe during the try-out regarding (a) 
what comments the learner might make (b) what reactions you might 
observe in the learner, and (c) what reactions you might observe 
from the Teachers* Guide. 

(1) . Arranging the try out: 



(2) . Conducting try out : 
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(3) . a. Possible learner conmients ; 



b. Possible learner reactions; 



c. Possible reaction to the Teachers* Guide 



You might wish to compare your answers with those suggested 
on page IV. Differences can be discussed during the institute 
group sessions. 



Suggested answers to hypothetical try’-out situation 2 



1 ) 



2 ) 



3 ) 




necessary try-out facilities Including A-V equipment j a 
"typical" 8th grade student must be obtained from the local 
schools to act as try-out subject ; arrangements for using 
the student are made through the superintendent; arrangements 
for car transportation and a schedule of pick-up and re- 
turn of student are made; extra copies of all Instructional 
materials and tests are produced. 



Conducting the try-out. The learner is briefed on how he Is 



to behave during the try-out; instruction is provided, 
carefully following all prescribed pedegoglcal details; 
notes of all student comments and responses are written 
into the materials at the precise point of occurrence; at 
conclusion of Instruction, criterion measures are admin- 
Is teredo 



m m ■ ^ ■ 

a. Possible learner comments. The learner Is apt to 
state that he cannot find the author’s name from the 
visual; it is possible he will state that he does not 
understand what to write in the worksheet. 

b. Possible Xeamer reactions . The learner might be 

observed copying into his worksheet all details from 
the visual without any apparent understanding; or the 
learner might appear confused and not be wjL'iting any* 
thing. 



ContM 3 



Ce ]P*088lble reactions to the Teachers* Guide . liie guide 
materiala do itot parallel the student materials. No 
explanation Is given for extra details such as author/no 
author, volume number, + sign. 



Stage (9): Another quality control point in developing an 

Instructional system Is to analyze the try-out of the product . 

The purpose of this stage Is to determine whether the Instruct- 
ional system attains its objectives. Two primary uses are made 
of this analysis: 1) to d^uermlne what portions of the new system 

are faulty or not adequate and require modification, and (2) to 
determine which specifications were unrealistically and/or Impro- 
perly established and should be changed. 

Analysis of the try out of a net^ Instructional system is not 
nearly so precise as that resulting from the customary experimen- 
tal research: The principal difference centers around controls. 

Where experimental studies place considerable emphasis on appro- 
priate controls, developmental research of new Instructional 
systems, on the other hand, seldom offers any basis for controls. 
The new development is just that, new I No suitable comparable 
thing exists against which it can be compared. Similarly, no 
substantial basis exists against ^Thich to control. 

Analysis must be maintained on an objective rather than com- 
parative basis. That is, how far does the system go toward 
achieving the ends for which It was developed rather than how 
much better did the system perform compared to some other system. 
When a new system has been sufficiently developed and refined and 
in use over a period of time such that its accomplishments can be 
predicted, then comparative analyses with another established 
system are appropriate, l^hen the system Is still new and essen- 
tially untried the only basis for judgment lies in how well it 
attains the objectives for which It was created. 

Other developmental research efforts also employ the ob- 
jective oriented analysis for making judgments. For example, 
much of the performance of the Boeing 727 In wind tunnel try- 
outs was analyzed against new objective standards. Although 
certain factors were common to other older systems, the complex 
combination of old and new had to be very rigorously tested 
against newly established standards. The only basis for accept- 
ance was successful attainment of the standards. If a portion 
of the 727 system failed in the test, careful analysis was made 
of what contributed to the failure so that modifications could 
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be made. The analysis often could not fix the exact cause of the 
fault, however, it did pin-point the area of weakness and thereby 
contributed significantly to plrnning for modifications and sub- 
sequent nevr trials. 

How, then, does one proceed in objective analysis? It is 
tempting to say all is fair in love and new instructional system 
analysis ana anything goes. However something better must be said 
if the concepts of research and the ’’scientific method” are to be 
amplified. The difficulty Is that the scientific method, in the 
form that it Is generally regarded, other than the ideal of rigor- 
ousness, has little guidance to offer this form of developmental 
research. We still knov; too little about how best to analyze the 
results of systems try-outs to prescribe any precise methodology. 
Literally, any and all means for analyzing results that meet the 
concept of rigorousness are currently being employed. 

Generally speaking, analysis must closely parallel try- 
out activities. That is, as small segments of the instructional 
system are tried out with various specialists and individual 
learners, analysis of these try-outs must occur in order that 
necessary modifications to the segments and next steps in the 
system development can be mad?^ 

Feedback from specialists can occur in various forms depend- 
ing upon the arrangements made. Discipline specialists will 
usually prefer to study the materials content privately making 
editorial notes of both a technical and stylistic nature. The 
developer usually reviews these notes with the specialist for 
clarification before analyzing their implications. Media special- 
ists will frequently prefer to judge the materials in concert with 
the developer to be sure of interpretations. Criticisms in these 
Instances are noted in detail by the developer and through this 
interaction take the form of analysis. Try-out with a typical 
teacher is usually jointly managed with the developer present. 

As in the situation with the media specialist, the analysis results 
from the interaction of the two. 

Analysis of sma‘*.l segment try-outs with learners depends 
upon the nature of the observations employed and the form which 
they are recorded. In many instances, the very fact that learners 
express having difficulty with a particular portion of the new 
system will cause the developer to view that portion in different 
perspective. What appear to be obvious verbal flaws or sequences 
sometimes almost "spring out" at the analyst as a result of such 
learner feedback. For example, analyze the Readers’ Guide 
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elements you hypothetically tried out in stage 8, Page V-21. 

It p>robably became immediately clear to the reader as a result 
of the learner *8 statement that he could not find the author’s 
name in the visual that the visual did not Include an author, 
yet the author was called for in the worksheet question. This 
form of analysis points to obvicus flaws in the development that 

9VU1I5UUW UVCJi, 4 -WWiC6u X&A UCUll*JUJtlJ.(3 V4*H=i WA. 

lead directly to revisions. 



Other forms of analysis are often less obvious. For example, 
perhaps the only evidence of weakness in an instructional segment 
is that each try-out subject has not learned a particular bit of 
information coming from a specific part of the instruction. How- 
ever, no obvious flaws, as above, are evident In the materials. 
Careful study of the segment might reveal, for example, that 
double meanings are possible in the choice of words included and 
have led learners to an erroneous interpretation, or that parti- 
cular sequential elements have combined to produce conditions of 
retroactive inhibition. 



Look again at the Readers* Guide example on Page V-21 and 
make a further analysis. Give consi.deration in your analysis to 
both the visual and vzorksheet portions and suggested answer 3 
given on page V-22. Briefly summarize your analysis in the space 
that follows, then check your answer with that given on page V-26. 



Analysis; 



ANSWER 

Analysis: The visual gives too much information at one 

tim^ perhaps separation of elements would focus learner 
attention more prec:^jaly to the desired portions. Work- 
sheet format does not permit the learner any routine or- 
der for writing down answers. This format combined ^/ith 
the too much detail in the visual results in learner con* 
iusion. Also, lack of author’s name in the visual con- 
fuses the learner. — 

m 



A more formal means of analysis than that described above 
occurs with criterion testing. Considerable attention to validity 
and reliability of the tests are necessary. Gagne, in describing 
research dealing with attainment of learning sets relevant to ter- 
minal objectives, actually demonstrates one example of how to an- 
alyze the try out of an instructional program using criterion 
measures (1961, p. 10). He first developed a hierarcy of learn- 
ing sets, similar to that displayed in section III of this manual, 
by employing the question *Vhat would an individual have to know 
how to do in order to achieve successful performance of this part- 
icular class of task, assuming he were given only instructions?" 
Five major levels of learning sets were thus formed, two levels, 

III and IV, being sub-divided once each. 

In testing the measure of transfer among learning sets Gagne 
analyzed four possible empirical relationships for passing and 
failing relevant hlgher-lox^er learning set combinations. These 
four were as follows: <1) ++: indicating positive transfer from 

lower to adjacent higher learning sets; (2"^ — : indicating fail- 

ure of lower level set followed by failure of adjacent higher 
level set; (3) +-: indicating failure of lower level set followed 

by passing of adjacent higher level set; and (4) -+: Indicating 

passing of a lower level set followed by failure of adjacent 
higher level set. 
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The patterns of pass~fail relationships on a criterion 
test administered following the learning program are presented 
In Table 1. (Gagne, 1961, P. ID) 



Table 1 



Pass-Fail Relationships of Achievement 

Higher Level Relevant Learning Setf 



^ ^^acent Lov;er and 



Transfer to i Frequeaey of pass-fall 

T-eamlng Set | Pattern— Higher , lower 



IV2 from IVAl 
llV5 from IVA3 
IIIAl from IV2, 1V3 
IIIA2 from IV4,IV5,IV6 
lllll from IVl 
}III2 from IV:5, IIIAl 
|lII3 from IVA2,IV3 
|III4 from IIIA2 

1111 from IV2, 1112,1113 

1112 from IVA2, III3 
|II3 from III4 

II4 from III4 

11 from III, II2 

12 from II2, II3 

13 from II3, II4 
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Although these results were primarily intended tor other 
purposes thev do demonstrate a good example of determining weak 
areL In a newly developed Instructional system. ^ 

in the interpretation of the total tabxe are referred to Gagne s 

monograph . 

In Table 1, the frequencies in the -+ column indicate the 
points In the instructional proRram at which greater or lesser 
effectiveness was attained. Large frequencies are interpreted 
as points at which the program was relatively ® ® 

frequencies the points at which learning was effective. For ex 
ampL, as shown in Table 1, progression from lower level l®a^‘ 
ing set- IVl to adjacent higher level learning set IIIl was failed 
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by 63 of the 118 learners. Examination showed that only one frame 
of the program was devoted to the content of set IIIl. Obviously 
insufficient instruction was provided these 63 learners in that 
one frame for them to fully understand the- new inforniation. Simi- 
lar results pertained to learning sets II4 and IIAl in which 24 
and 26 failures respectively occurred after successes at the lower 
level set. Depending upon the level of errors acceptable by the 
developer, other frames in the program in this example would be 
subject to similar scrutiny. This form of analysis obviously re- 
lies very heavily upon a powerful criterion test— one that tests 
all aspects of the Instruction system. 

Equally as crucial to the analysis of the instructional 
material try-out is analysis of the measuring instruments. Only 
as test items are developed, tested and modified until valid and 
reliable for measuring changes in behavior resulting from the in- 
structional system can there be confidence that desired learning 
outcomes are actually being achieved. This step is mentioned here 
only to give it proper continuity; further discussion of it i^ll 
be given in section IV. 

Stage (10) ; As try-out and analysis of the new Instructional 
system progresses, the purpose of all this is to provide a sys- 
tematic basis for instructional material modification . During 
this stage, results of the analysis are translated into changes 
in the product. In some instances the revisions vrill virtually 
be known to be an improvement, in others only additional try-out 
will sc determine. For example, analysis such as the omission of 
the author’s name in the Readers’ Guide materials should prompt a 
modification In the visual quite certain to improve that aspect; 
hovjever, revisions to the visual and worksheet based on the addi- 
tional analysis cannot be mvade with such confidence. In both cases 
they must be tried out to make certain that the modifications 
are effective. This latter is a crucial point to the successful 
completion of an effective product. 

Our Boeing 727 analogy might strengthen this point. For 
example, early test flight trials of the 727 revealed that at a 
certain angle of climb, one designed for takeoffs, the engines 
lost significant power; however up to that angle no power loss 
was sustained. Analysis determined that the air intake housed 
in the vertical tall structure was placed too lov; to receive 
sufficient air at the critical climb angle. Modifications were 
made placing the air intake higher in the tail structure. How- 
ever, the 727 remained unacceptable until additional flight tests 
were completed at various angles of climb to determine whether 
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takeoff objectives were being met. 



Modifications should be planned with the same team of special- 
ists who contributed to the development. For example, consider 
the Readers * Guide elements introduced on page V-21. Analysis 

chat the visual contained too much information at one time. 
Methods of reducing the amount of information should be determined 
in conjunction with the media specialist. 

\^at suggestions do you have for changing the visual? Staa- 
matize briefly In the space below the modifications you think 
would make the visual more effective. Check your answer with the 
one given on page V-30. 

Suggested Modification ;; 
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ANSWER 



Suggested modifications ; rne visual should ba 

from a single transparency into a base transparency 

with four overlays. The base should contain the title: 

nvi»vru»uu.i.\;o • rixu—axi. ^a/OjLo. 5 jlxx.ou uvex.xay uu uOii— 

tain the abbreviation for illustration: 11; the second 

overlay to contain the magazine name: Tima; the third 

overlay to contain volume number and pages; 93:62+; 
and the fourth overlay to contain the date: J1 21 *67. 

In this manner, information can be presented in smaller 
segments and more control is thereby available to direct 
the learner’s attention to the specific element being 
presented . 



In conjunction with the revisions recommended for the visual 
the analysis suggests that other changes be made to the Readers * 
Guide example of instructional materials. The worksheet example, 
from page V-21, should be modified into outline format with each 
bit of information separated as follows: 

On the screen is a sample of an entry from the Readers * 

Guide . In the spaces that follow copy the information from 
the visual explaliiing the meaning of each of the following: 



1. 


Title of article 




2. 


Author (if given) 




3. 


Type of illustrations (if any) 




4. 


Name of magazine 




5. 


Volume and Page number 




6. 


Month, day (if given) and year 





It is further suggested that a second visual be prepared 
containing a Readers * Guide entry having an author and other 
symbal variations not contained in the first sample entry. In 
addition ♦•.he Teacher’s Guide shown on page V~21 should be modi~ 
fled to present the information to the teacher in a form that 
parallels the student worksheet form. Explanatory information 
should be included for the teacher regarding placement of author’s 
initials, the fact that some entries are without author, use 
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of color separating volume and page, reason for volume numbers when 
magazines are bound, meaning of the +, and abbreviations used in 
the date. 

Stags (11) : The final stage of developing the new Instruc- 

tioriX systssi concfiims tilis of sXX product d^vcXops^nt 

stages. That is, returning to stage 7, producing the instructional 
product, and going through the cycle again. The purpose of this 
stage is to point oat the importance of continuing with the develop- 
ment, testing, analysis and revision process until the desired 
outcome learner i^ehaviors are achieved. 

Obviously this total process is not one of a series of four 
discrete stages of which the second cannot be started until the 
first is completed, etc.; rather, it is a complex process of over- 
lapping and combining. It Is not unusual for different stages to 
be occurring together; i.e., a trade back and forth between pro- 
duction, try-out, etc., various segments of the system. It 
might be more efficieut^ for example, to develop several similar 
visual elements at the same time even though they are located 
in different parts of the system. Feedback regarding their fidelity 
and readability can be obtained before the instructional sequence 
to which they belong is ready for trial, thus pushing towards time 
and cost efficiency. 

To the extent the developer can perserve in the development 
sycle until desired behaviors are achieved his product will im- 
prove; assuming, of course, that the desired learner behaviors are 
Indeed desirable. 
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ERIC 




Evaluation Sheet 



yie need to know how well the ideas and issues in this manual are 
communicated to you. You are the test audience for this material. 
To remove or strengthen the weak spots, to retain or Improve the 
strong ones (if any), an account of your learning experience as 
you read these sections Is crucial. Use this sheet as you study 
and jot down your reactions. Don’t be concerned with typographical 
errors. We’re concerned with how the message Is coming through. 

SECTION # TITLE 



Give topic, 
paragraph or 
sentence, page 



Importance 

Rate 

1 2 3 4 5 
Low - High 



Clarity 

Rate 

1 2 3 4 5 
Low -High 



Suggested 

Improvements 



Over-all Rating of the Section: Use a 5-point Scale 



Impo rtant 
1 2 3 

Very 
Low 

(Circle one) 



Very 

High 



1 

Very 

Low 



Understandable 

2 3 4 5 

Very 

High 

(Circle one) 
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Section VI 



Research Design 
John Gordon, Jr. 



Overview 

This chapter deals with the fundamental logic and logistics 
of Instructional research. When you have completed the session 
we hope you will be able to do a ’’first level" analysis of a re- 
search proposal and provide some constructive comments. We be- 
lieve analysis of other proposals to be the first step toward 
constructing your own design, but unfortunately as all of us 
know one task Is only prerequisite, not supplementary to the 
other. 

Experimental Design 

The many definitions of experimental design In educational 
research range from the general plan of attack to the specific 
method of assigning the experimental treatment to the experiment- 
al unit. For the purposes of tfels institute we'll stay with the 
latter and be as specific as possible. The ^cperlmental treat- 
ment is defined as those conditions the experimenter has decided 
to m^ipulate in a specific way. This rather gross category is 
also referred to as the Independent variable . The experimental 
unl^ in our case in instructional research is usually the stu- 
dent or that unit which is being acted upon by the conditions of 
the experimental treatment. There are also times when it becomes 
necessary to consider the entire class unit as an experimental 
unit. Using the entire class as an experimental unit is typical 
in instructional research in colleges. Another way of locating 
the experimental unit is to look for that which will be measured 
to determine whether or not the experimental treatment has been 
effective. These measures taken on the experimental uc ^ are 
also called the dependent variables . 

Again, experimental design is the "method of assigning" the 
experimental treatment to the experimental unit. The most common 
method is called random assignment , a typical example of which 
is drawing numbers or names from a hat. The fundamental assump- 
tion underlying this method is that every experimental unit has 
an equal opportunity of receiving the experimental treatment (s) . 



The usual procedure ror random assignment is to use the table 
of random numbers that is found in most statistics and design 
texts* The procedures are spelled out in each text but gener- 
ally? 1) allots numbers to the experimental units, 2) picks 
any number la the tablet 3) continues down or across that row of 
numbers matching those that are in the table with those that have 
been allotted, and 4) put them in either the experimental group 
or other groups Another way of looking at the equal probability 
assumption of random assignment is to consider it a way of elim- 
inating systematic bias* A systematic bias is defined as any 
condition, other than treatment conditions, either within the ex- 
perimental unit or in the environment or context of the experi- 
mental unit that would be present for s^ll members of one group 
and not another* This undesirable co3^\ditlon would also have to 
have a measurable affect upon the dependent variables or those 
measures of the experimental unit which you are hoping to change* 
Randoft assignment is considered, by most, as the optimal way of 
designing an experiment. Other methods are at best, poor sub- 
stitutes* These methods, suc^ as matching, color balancing, and 
statistical control, e*g, covariance analysis, all have funda- 
mental problems. Such methods' are considered more fully in de^- 
sign texts and will not bt cohered here* 

Another basic distinctipn made in most discussions of design 
is that between uaivarlatjB. find multivariate experimentation. 
Univariate meaps tj^t one' particular conditutlon is varied and one 
particular measurement thougj^t to be associated with the condi- 
tion is assp^sed, Foy eJtople, a movie (experimental condition) 
may be showp to some etpdenps and not tc others and a knowledge 
test (simplis neasupe) ai^nlstered after having witnessed the 
film. Hult^-Variatp experimentation is based on the notion of 
multiple causation whlclj.' ilssumes that a combination of variables 
or experimental treatments acts upon a number of dependent vari- 
ables pr 4ifierent measures taken of the experimental unit* For 
example', th® movip may b^e shovm to a certain kind of student and 
this combl^ition of stujdjiet and film influence the student in a 
number oF li^ys, e*g*, fpt both attitude change and knowledge 
content f Apy combinatipp sequence of independent variables 
or treatments and depe^^ent variables or measures is classified 
as multivariate o^oritnentatlon* A piece of advice for the be- 
ginning instructions^ researcher would be to attempt a univariate 
form of efqierimept before considering the nultivarlate although 
the latter, are gimp^ally considered to be much more valuable and 
representa^ve of rpallty. 

i 

Your npjjor task in this set of self-study materials will be 
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an analyais and critique of two proposals that are Included In 
the latter part of the materials. The following is a check list 
to be used as a guide. 

1. What is or are the experimental treatments? 

2. What is o? are the experimental units? 

3. What method of assigting the treatments to the units has 
been used, in other words, what designy 

4. Using the designs 1“6 from Campbell an^ Stanley, deter- 
mine in which category the particular ppoposal design 
falls? (Determine the R’s, X*s and O's) 

A more important guide, however. Is the qpjaprehensive cha|^ 
prepared by Drs. Donald Campbell and Julian ^b^nley which appears 
in the Handbook of Research on Teaching. Tlie- ^har^. and def Ini;* 
tiotts are the crux of an excellent discussion .^htch you may wish 
to read and refer to in the future. The chart is shown in 
Table 1. 

Once you have judged what design category is being used you 
are then able to move directly to the chart to assess the sources 
of invalidity of the particular design. Starting with the 
strengths (+’s), you search out sentences in the proposal which 
show that the author has made note of these possible sources of 
invalidity and how this design overcomes them. If the author 
has not made note of the strengths of his design then he is in 
error and should be amended. More Important, however, is the 
determination of the weakness (es) of the design. The author 
should emphasize these weaknesses and his attempts to control or 
compromise these sources of error. Tlie argument should be clear 
and strong as to why certain areas or sources could not be con- 
trolled. For example, study of a short duration would not have 
the maturation effect as one of its concerns and the proposal 
writer should make note of this. In other words the argument 
should be clear, highlighting those major sources of Invalidity 
controlled either by the design or other means peculiar to the 
experiment, and those minor sources not thought to be of Import- 
ance or at least compromised. 

It now becomes imperatl/e that we look more closely at the 
sources of invalidity stressed by Campbell and Stanley in their 
classic article. A list of these sources and their definitions 
can be found in Table 2. But if, for some reason, these defini- 
tions are not understandable we will attempt a short synopsis at 
this point using common examples. Read their definition first 
and this example second, if needed. 
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T^le 1** 

Sources of Invalidity for Designs 1 Through 6 



Sources of Invalidity 



Internal 


External 


History 

Maturation 

Testing 

Instrumentation 
Regression 
Selection 
Morality 
Interaction of 
Selection and 
Maturation etc. 


1 Interaction of 
Testing & X 
Interaction of 
Selection & X 
Reactive 
Arrangements 
Multlple-X 
Interference 


Fre-E]qperliiKatal I^eslgns: 

1. One-Shot Case Study - - - - 

X 0 


- 


2. One-Group Pretest- ----? + + 

Post-test 

0X0 


- - 7 


3. fta'*tc-Group Comparison +? + + + -- - 

X 0 

b 


** 


True Experimental Designs: 

4. Fretest-Poet-test *^ + + + + + + + 

Control Grouo Deslen 


- ? ? 


5. Solomon Four-Group + + + + + -h + + 
Design 

R 0 X 0 
R 0 0 
R X 0 
R 0 


+ ? ? 


6. Poet-test Only Control + + + + + + + + 

Groiqp Design 
R X 0 

R 0 1 


+ ? ? 



* Hlnuft Indicates a wejkness; plus, a factor controlled; 

question, a source of concern; blank, an Irrelevant factor. 
These are rough Indicators only. Complex designs and quali- 
fying conditions nay alter the Indicated effects; 

** Abridged from Chapter 5 "Experimental and Quasl-experlmental 
Beslgns for Sesoarch on Teaching." Handbook of Resear ch on 
Teechlna (1963) • 
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Sources of Invalidity 



^^8toi ’v« the specific events occurring between the first and 
second measurement In addition to the experimental treatment. 

2. Matwatlon. processes within the respondents operating as a 
function of the passage ox time per se (not specific to the 
particular events) » Including growing older » growing hungrier, 
growing more tired, and the like. 

Testing^ the effecta of taking a test upon the scores of a 
second testing. 

4. Instrumentation, in which changes In the calibration of a 
measuring instrument or changes In the observers or scorers 
used may produce changes In the obtained measurements. 

5. Statistical regression, operating where groups have been 
selected on the basis of their extreme scores. 

6. Biases resulting In differential selection of respondents for 
the comparison groups. 

Eroerlmental nortalltv. or differential loss of respondents 
from the comparison groups. 

8. Selection-maturation Interaction, etc. , which In certain of 
the multiple-group quasl-experimental designs, such as Design 
10, Is confounded with, l^e., might be mistaken for, the 
effect of the experimental treatment. 

9. The reactive or Interaction effect of testing. In which a 
pretest might increase or decrease the respondent’s sensi- 
tivity or responsiveness to the experimental treatment and 
thus make the results obtained for a pretested population 
unrepresentative of the effects of the experimental variable 
for the unpretested universe from which the experimental re- 
spondents were selected. 

10. The interaction effects of selection biases and the experi- 
mental treatment. 
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H* Reactive effects of experliasnta'i arrangenents. which would 
preclude generalization about the effect of the experimental 
treatment upon persons being exposed to It In nonexperiinental 
settings. 

12. Multlple^trestment Interference, likely to occur whenever 
multiple treatments are applied to the same respondents^ 
because the effects of prior treatments are not usually 
erasable. This Is a particular problem for one-group de- 
signs of type 8 or 9« 



But iff for some reason, these definitions are not under- 
standable we will attempt a short synopsis at this point using 
common examples » Read their definition first and this example 
second, if needed. 

1. History - It is not uncommon for experimenters in in- 
structional research to forget that the textbook Is 

a study material which suppliments any lecture or 
movie which may be the treatment and that some students 
possibly either in the treatment or control groups 
have access to the textbook. It is extremely difficult 
to control for outside studying In instructional 
research. 

2. Maturation - Long term curriculum studies which observe 
students over time periods in excess of three years 
find great difficulty attempting to control the 
effects of maturation. 

3. Testing - Providing students with an exam which gives 
some indication of the outcomes of instruction usually 
Increases the IR scores on post test. In attitude 
studies the administration of a pretest usually 
depresses the intensity of the attitude change. 

4. Instrumentation - It is common in studies where a 
number of ratings of teachers or students are made 
over a period of time that the raters become more 
proficient and therefore change as a measuring Instru- 
menter. Frequent checks of the reliability of the 
rater and his accuracy in making the Judg^oents should 
be planned for In the design. 

5. Statistical regression - Theorists many times concern 



tUemselves with a particular group of ludividualSy a* 8* 
IQ. Uafortunately In multiple testing of -these 
extreme groizps, their scores statistically regress to- 
ward the mean and wash out possible treatment effects. 

Any attempt at random selection of extreme groups, there- 
fore should be carefully considered in the design. 

Biased selection - until recently it was not uncommon 
to find subjects being matched on a series of variables 
thought to be highly related to the treatment variable. 
Unfortunately, 1) all of these selection va iables 
crxnot be controlled, and 2) the loss of students who 
are unmatchable is large an^ unestimable. 

Experimental mortality — Sickness and dropouts from 
classes tend to diminish the conclusiveness of the 
results of any experiment. One way to prepare for 
these contingencies is by having a large sample. 

Interaction of Selection and Maturation - There are 
occasions when one wishes to use the entire sophomore 
class within the experiment, seeks to find a coaparable 
group and the best he can locate is a junior class. The 
experimenter runs the risk of this particular source of 
invalidity since the juniors with their year of growth 
may in fact be different than the sophomores. This 
maturation may produce a gain that can easily be 
m is take n for the effect of the experimental treatment. 

Interaction of Testing and Experimental Treatment - A 
pretest has the general effect of increasing scores on 
a cognitive test and decreasing scores on an attitude 
test. It can be easily seen that a pretest is a 
practice of the post-test. Also it provides the student 
with a set toward the learning material in the sense 
that he knows what is to be €uq>ected. In an attitude 
study the pretest also has ’’set*' to introduce the 
student to the particular attitudes thsit the film or 
whatever the experimental treatment wishes to influence* 
The student is presumably much more conscious of the 
manipulation and leas likely to change in the desired 
direction. 
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10« Interaction of Selection and E3q;>eriaientdl Treatoient - 
In nany experiments students are asked to volunteer 
their services as subjects* This seemingly unimpor* 
tant fact that the students volunteered may interact 

wl Irfi f*‘ho atil 

VM i#Jbao WHO ^t?oaxC9s 

Here again, this source of invalidity demonstrates the 
tremendous problem one has in selecting students by 
matching or some other way other than by random * 
assignment. 



11. Reactive Effects of Experimental Arrangemei^td • It is 
extrem&'ly difficult to remove this source of Invalidity 
since every student usually is aware of the fact that 
he is a meod>er of an esqHeriment. Wa ajj try for the 
"candid camera" effect but many tiites it is impossible 
to bring off. 

12. Multiple Treatment Interference r In the more complex 
es^eriments we find that when one has multiple treat- 
ments he sometimes wishes to try each of these treat- 
ments on every stud^t, thus encountering a host of 
possible errors, e.g., the effect of the first treatment 
upon the second and so on. 



Now read Practice Exercise 41, As you read the proposal, 
answer for yourself the following four questions: 

What are the eiqterimental treatments? 

What are the experimental units? 

What is the method of assignment? 

What design category does this project seem to fit? 

You will note that these are > the same four questions given above 
on your checklist. After you have answered each question you may 
coiqpare your anst^er with those given below. 

Answers to Practice Exercise Number One . 

What are the experimental treatments? 

ANSWER 

Although the experimental treatments are given 
a heading, a closer inspection reveals them to he 
terribly complicated and multi-faceted^ This is a 
typical case of too many eggs in one basket, and if 
certain effects aren't forthcoming there is no way 
of determining which of the number of different 

I treatasata has caused which effect. 
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What ara the experlnental units? 

AHSWER 

At first reading It seems that four groups are 
going to be considered the exper i me n tal units, but 
further on we find that subjjects or stud^ts will be 
assigned to each of these treatments > 



What is the method of assignment? 

AMSWER 

It is obvious both that some form of matching 
is hinted at but not explained ^ herein 

lies a second fundamental flaw in this proposal. 



What design category does this project seem to fit? 

ANSWER 

In searching for R's, X*s and 0*s, ye find first 
that there is no random assignment but some form of 
matching. Second, all groups receive one kind of 
treatment* Thirdly, there are weekly observations 
on each student* Having more than one measure on 
each student, as you can well see, further compli- 
cates this already complex proposal* A designation 
of this study in one of the six categories would 
depend primarily on the use of the observations* 

If the first Monday-morning test were to be used as 
the pretest, then it is possible that they look like 
a group 2 deslgc^, where each of the experimental 
treatment groups Is Its own comparison. 



If we quickly run through the sources of invalidity of 
this particular proposal we find tha . history is alluded to in 
the selection process but not really specified in the detail needed 
for this kind of study* Maturation is not of too much concern 
because the study duration is only one term* Each group or each 
individual in each group receives a test Monday morning therefore 
somewhat controlling for the effects of testing upon further 
tests* (InstrumeRtation) Raving a single rater score each of the 
student responses especially with his own numerical, scheme is wide 
open for criticism* If a rating scheme is used then a number of 
raters, possibly four or five should be used and tbelr results 
correlated to determine their reliability and consistency over 
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time. The writer says nothing about selecting groups on the 
basis of extreme scores, although the selection criteria are not 
well spelled! oat in the first place, so the statistical regress- 
ion cannot be considered a fault* The next source, number six, 
biases resulting from differential selection * is again an unknown 
but obviously o tremeXidouS problem lu this proposal* No mentiou 
is made for any attempt to control experimental mortality or stu- 
dent dropout or sickness* In small classes such as these, fifteen 
it becomes crucial then if students stay away from class for a 
couple of days and lose their * academic place * In the course of 
an entire semester there is no telling what effect this will have 
on the overall results* Again, when considering the dbiteraction 
of sslgction and maturation the lack of apecif icity in detailing 
the selection techniques makes it impossible tc make a judgement 
in this case* The last four sources of invalidity are determined 
primarily through interaction with the experimental treatment* 
Since the experimental treatments are not well defined then it 
becomes impossible to make judgments on these four as well* 

One can tell in reading this study that it is a typical 
instructional situation where an Innovative set of ideas is try- 
ing to be implemented into the present instructional system* 

This is exactly what we are attempting In developing better, more 
challenging college instruction* Yet the proposal is a perfect 
example of the tremendous conq>lexity of college Instruction and 
the difficulties in doing lo"^g, term course length, studies where 
so many intervening types of activities > come between the student 
and the total effect upon him of the course* The best advice we 
can give to this proposal writer is to make much more specific his 
experimental treatments and attempt to assign some theoretical tie 
between them and the tasks that are being learned* With this the- 
oretical tie made he can make some fairly specific hypothesis and 
tie the study down in terms of time and complexity to shorter 
periods, perhaps a day or a week where short term studies of thw 
effects of these variables can be culled out. The writer has the 
making of these smaller studies when he considers the weekly test- 
ing. He might capitalize on these weekly tests in an attempt to 
show differential gain over a period of time rather than the one 
lump sum approach. 

Now read Practice Exercise //2. As you read the proposal, 
answer for yourself the following four questions: 

What are the experimental treatments? 

What are the experimental units? 

What is the method of assignment? 

What design category does this project seem to fit? 



You %rlll note that these are the saise four questlosu* given above 
on your checklist. After you have answered each "4 testion you |^y 
compare your answer with those given below. 

Answers to Practice Exercise Number 

What are the experimental treatments? 

ANSWER ■ ■ ■ 

One of the treatments obviously is the tisin^ of the 
electronically modified piano. It is not stated v^t the 
subject or pupils will do with this piano but we can assume 
that he plays it axid it plays back to hiie somehow. A much 
more difficult problem arises in the notion of conven- 
tional Instruction. Entirely too many studies rely upon 
this gimmick to back away from explai^ng wha^ act^ially 
does happen to these students. It Is relatlvsly obvious 
that the writer doesn^t know and each student 'probably 
receives some different instruction'. The writer has tried 
to camouflauge this by saying that it is an ihetructional 
method widely used throughout the country. 



What are the experimental units? 

ANSWER 

7 It is obvious that subjects' or students are the experi- 
mental unit* But it is not so obvious that memjerb of one 
group receive the same trea^ent as members of the other. 

For example, each student seems to be studying independently 
of the other sutdents. Therefore one may study consider- 
ably longer vhile another student may be studying one key 
and not another key. Is it correct then to assume that each 
of these groups of students are receiving ''the" experimental 
treatment? - . . . 



What is the method of assigrment? 

ANSWER 

Here the writer Intimates that students will select 
dif f(irent sections of the course by chance by saying that 
"it is reasonable to assume that no factors other than 
chance are operating in the assignment of individuals to 
groups." Unfortunately, that is not an easy assumption 
to- accept for the single time allotment to each section 
mai^ times determines the type of tudent entering each 



Continued 



A^iSWER (Continued) 



of the sections* Again* anything less than random assign* 
ment lays the writer open to much criticism. The writer's 
reference to the further expectation that rotation of 
treatments will compensate for any differences in assign* 
meat to groups only confuses the matter more. If a 
student increased* let's say 10 points in the first 
group* what Is the effect of that Increase on the second 
is an experimental question, not one that easily eliminates 
itself. 



What design category does this project seem to fit? 

ANSWER ■ 

Again, this study, like the previous one is categorised 
somewhere between three and six. Since there was no true 
randomization and only one observation made after the treat* 
ment we have to put It somewhere between these two. 



Let us run down the list of sources of invalidity. As was men* 
tioned earlier, what happens between that first test and the last 
is anybody's guess* When the treatment is unspecified, history 
effects also are under question. As in the last study, maturation 
effects are fairly well controlled since each of the groups mature 
together. The question of testing effects becomes somewhat com* 
plicated when one considers all the testing that is done during the 
course Itself. The instrumentation problem is handled with sophls* 
ticatlon in this study with three raters, each of which does not 
know the identity of the student. Indices of inter*rater rella* 
billty are to be determined and reported. Statistical regression 
effects don't seem to be of concern because extreme groups are not 
in use. Again, the problem of selection is left somewhat in doubt 
because of the gross assun^tlon that students electing courses 
will be the same as a random assignment. With only 20 students 
per class the experimenter is flirting with danger wh^^n not con* 
sidering the mortality problem. The Interaction of selection 
and maturation remains a problem as the selection procedures are 
somewhat under question. Since the pretest was simply an Initial 
proficiency test it might be argued that there would be no inter* 
action with the treatments although one cannot really say. The 
selection of experimental treatment effects again are based upo'tx 
the weak assumption underlying the selection procedure. The 
reactive effects of the experimental arrangements is an extremely 
problematical area In this research. Those using the electronic 
keyboard certainly know tlmt they are being watched, so to speak, 
and those not using the electronic keyboard are envious of those 
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who are* How one controls for these reactive effects is \tsnally 
throng sons form of counterbalancing, giving A then B for one and 
B then A for another* As in the first study, breaking the experi- 
ment up In much smaller parts and giving one student a try at the 
piano with certain conditions* and another student at another tlise, 
each student is affected by each treatment* But in doing that one 
falls into the obvious pitfall of the last source of invalidity; 
mainly, the multip le treatment interference . It is entirely 
possible that the student who begins the electronic keyboard may 
become very depressed with the outmoded conventional technique* 

In this experiment as the one that came before It, the 
experimental treatments were not consistent over individuals in 
the experimental and control groups* Therefore, it might be well 
to proceed on an individual basis recording the rate of learning 
of each individual and perhaps through counterbalancing of treat- 
ment end conventional or electronic and nonelectronic instruction 
one can determine the effectiveness of one over the other* Again 
a caution must be made in making the task too large, that is, con- 
sidering an entire semester's course of study. It would be much 
more appealing if one t:ould consider each of the tasks, perhaps 
scales left hand, right hand, or any way of breaking down the 
total task into smaller units and looking at the effect of the 
keyboard on the smaller units of study. using the smaller 
\mlts and possibly Interchanging students ..rom one instructional 
method to another, it would then become possible to remdomly 
assign students to treatmesits rather than have to be constrained 
by the ad^dnlstratlve scheduling of the courses* Again, it seems 
that this type of study could have been much better carried out in 
a one to one basis or a laboratory approach where one student or 
each student would be watched carefully over a period of time to 
determine the influence of either of the two methods* 

With the time remaining, read Glass Exercise #1 (Appendix C)* 
Answer fox yourself the same five questions for this proposal and 
be prepared to discuss the proposal during class* 



Appendix A 



Practice Exercise IX 



A Study of the 0se of Motion Pictures 
In Foreign Language Instruction 



!• Problem ; 

Recent survey® reveal extensive numbers of foreign language 
laboratories In use In the schools colleges, and universities of 
the United States (Social Science Research, Inc., 1963). . 

most of these laboratories are being used Ineffectively Is well 
documented (T.C., Columbia Univ,, 1963). The purpose of this 
proposed research is to investigate a new method for increasing 
the effectiveness of language laboratories. The method to be 
Investigated makes use of motion pictures, and was designed as a 
result of accidentally obtained pilot data which indicated the 
feasibility of using motion pictures to enhance language labora-* 
tory effectiveness. Tlie incidental finding resulting from delayed 
receipt of materials suggests that motion pictures will enhance 
'She effectiveness of the language laboratory to a significant 
degree If they are used following the language laboratory 
experience but will impe^ . learning If used in conjunction with 
the laboratory experience. There are good theoretical bases for 
the position. 

IX. Related Research ; 

In the investigation conparlsons will be made of the per- 
formances of students following three types of learning experi- 
ences. Students in a course in Modern Spanish will be studied. 
Selection of Modern Spanish was seen as unique because (1) the 
content has been agreed upon by experts in the field and (2) well 
developed materials are available for use. The use of the visual 
materials proposed in this' study is quite diverse from that 
reported by Creore and Hanzeli (1960). The method reported by 
Creore and Hanzeli will constitute one of the controls in the pro- 
posed experiment. A more complex design than that used by 
Creore and Hanzeli will provide a more adequate test of hypothe- 
ses. pertaining to the use of films. The proposed use of visijl 
materials also differs from that enq>loyed by Asher (1961) in which 
subjects leaxu^ a Spanish vocabulary list using one type of 
stimuli Cvisual or auditory) and relearned the list using the 
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other stloulus mode. Because of the divergent use of the visual 
stimuli by Asher, no control of his techniques will be made in 
this study. 

EKtessive research in the ^acquisition of foreign language 
skill has resulted in numerous guidelines for foreign language 
instruction (c.f. Modem Language Association, 1961; Lane, 1961). 
Little evlderce is available, however, regarding the appropriate 
use of motion pictures In the foreign language classroom or 
laboratory. 

There have been mimerous uses of motion pictures in Instruc-* 
tion. Films have been used to introduce units of study, to 
present information, to motivate thought and activity, to modify 
concepts and attitudes, to Integrate a series of facts, ad 
infinitum . All of these uses ml^t be readily applied to foreign 
language instruction. It Is the purpose of this study to 
investigate three possible uses of a set of motion pictures in a 
beginning Spanish course for college students. 

The Spanish Course . Following suggestions of the Modem 
Language Association of .America (MLA), the faculty of Southern 
Oregon College (SOC) now offers an elementary course in Modern 
Spanish. The primary objectives of the course are to teach 
students to speak and to understand conversational Spanish. 
Behaviorally the objectives might be stated as follows: 

Following the course of instruction the student: 

1. Responds with correct grammar, tone Inflection, 
and accent, in Spanish, to a series of Spanish 
patterns, expressions and questions. 

2. Converses in Spanish with another individual 
about any of a number of common persons, places, 
or things including ideas. 

In other words, the course is intended to prepare a student 
to the point that he could express himself adequately and without 
language related eabarrassment in the Spanish speaking community 
such as Mexico City. It is not the primary purpose of the course 
to train students to read or write Spanish. This ii< consistent 
with the suggestions of MLA that students first develop spepking 
and comprehension fluency and later reading and writing profi- 
ciency. 
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A set of materials, including a text (MLA, 1960), magnetic 
tapes, and a aeries of correlated motion picture films, vhich were 
prepared under the auspices of MLA will be used In the Spanish 
Course at SOC. The text is divided into 30 units, of which 24 are 
developed around a conversation (dialoi^e) between two or more 
persons about tbe same topical episode* Each dlalcgue has 
associated with it a tape recording of the dialogue (conversation) , 
for use in a language laboratory, and a motion picture f 13m 
depicting the dialogue* MLA suggested that the dialogue tapes 
aid films be used for introductory experiences in the classroom 
and that the laboratory tapes be used individually by the students 
for practice* 

Upon initiation of the course at SOC in September, 1962, the 
films were not available* When the films arrived in October, 1962, 
the instructor began to use them in two ways— (1) as| an intro- 
ductory experience for future dialogues and (2) as a review 
experience for dialogues which had been previously completed with 
the use of the dialogue tapes but without the use of the films to 
accon^any the dialogues • Informal evaluation of student learsiog 
indicated that use of films seemed to result in differential amounts 
of learning and retention* The evaluatidn further suggested that 
learning was more complete for those dialoguas in which the fi^ 
was used for review than for those in which the films were used as 
an introductory experience* It is felt that a thorou^ Investi- 
gation of these informal findings Is merited* 

Theoreticsl Bases * The use of motion picture films as a 
review or culminating experience is consistent with behavioral 
theories which state that before a response can be used in appro- 
priate context it must first be in the repertoire of responses of 
the individual* This suggests that the development of a response 
repertoire in Spanish by the learner precede his attempts to use 
the responses in a meaningful fashion* Lane (1961) stated that 
the development of the xeeponae repertoire results from the 
application of principles of operant conditicnlng in the learning 
process* The activities in the classroom and language laboratory 
are roughly analogous to this type of conditioning although Lane 
and other Skitmerian psychologists would probably fiM the initial 
responses required in typical Spanish cla8*3rooms and laboratories * 
more complex than they would wish* In other words, after the 
responses have been acquired, they will then be associated with 
a meaningful situation* This arrangement, however, is only one 
approach that will be investigated* In the proposed study the 
filmed versious of the dialogues will be used to provide meaning— 
ful experiences containing a large number of vistial stimuli, to 
any one of which the verbal response may be associated* Tape 
recorded versions of the dialogues will also be used to provide 
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this experience; however* it is hypothesized that appropriate use 
of the vlsiial stimuli presented in the motion pictures will 
enhance the meanlngfulness and the retention of the total 
learning experience. 

A second experimental use of motion pictures will employ 
them as an Introductory experience. This application has as its 
theoretical base a Gestalt explanation of the learning process 
(Hilgard, 1956) » This orientation suggests that learning is more 
efficient and lasting when it occurs within a field which is 
structured either by cr for the learner* The total or whole 
experience is assumed to be a more meaningful one for the 
learner and one to which his responses may be related readily* 

This, apparently* was the basis used by Creore and Hanzell in 
their experiment. Unique "control” conditions will be provided 
to help clarify the meaning of differences in learning which are 
expected* 

III . Objectives : ^ 

The principal objective of the proposed study is to pro-^ 
vide evidence regarding procedures to be followed in enhancing 
the use of language laboratories. 

IV. Procedures : 

Pour groups of 15 students will participate in th^. study. 

Each group will receive instruction in a modern Spanish course 
under different treatments. The treatments are described below. 
Assignment of subjects to the experimental treatments will be 
done in such a way that the four groups do not differ in aca- 
demic ability, achievement level* or previous experience with the 
criterion foreign language, namely Spanish* 

ExperimePtal Treatments . The four treatments are shown in 
Figure 1. The treatments are so designed that amounts of tiro 
spent by learners in the classroom and laboratory are the sai^ 
regardless of the treatment. Study from text by the subject will 
not be controlled, however, each subject will be required to record 
and report the amount of time spent in this way* The . experiment 
will be conducted during the first academic quarter of the course 
which extends over three quarters. In this way greater control 
over previous experience may be maintained. Each group will meet 
in class on Mondays and Fridays for one hour. On Mondays each 
group will be tested on the previous week's assignment, and will 
receive an orientation to the week's activities. Orientations 
will differ by treatments. Fridays will be devoted to differ- 
ential review experiences which will be described in detail below. 
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students fdll be required to spend five hours In the l^nguege 
Itboratory between Uonday and Friday, and must spend at least one 
hour but no more than two houre each day in the laboratory. The 
laboratory practice will be divided into two phases. A basic 
phase (D) will consist of practice on the actual dialogue to be 
mastered. Once this phase is perfected, each student will prac- 
tice on an enrichment phase (E) Which provides practice with 
variations of the content In the basic dialogue. Experience 
indicates that on the average each student will practice approxi- 
mately two to. three hours in the dialogue phase and two to three 
hours on the enrichment phase. Attendance in the laboratory will 
be required and will be limited to five hours per week. 

The experimental design, thus, comprises four groups of 
students who will rscelve five hours of individual training in a 
language laboratory and two hours of class experience each week. 
The groups will be differentiated only in terms of differential 
experiencee they receive in the classroom setting. The experiment 
will continue for one academic quarter. 

Since the laboratory experiences for each treatment group 
do not differ, the description of treatments below will be 
restricted to the experiences received by each group in the class- 
room. 



Treatment A. On Monday of each week, the subjects in this 
group will be tes^ted over the training received during the pre- 
vious week. This will constitute about IS minutes of the class 
period. Following the test period, the subjects will be intro- 
duced to the dialogue to be studied during the week. Introduction 
experiences for this group will consist of listening to several 
repetitions of a tape recording of the dialogue. After each 
repetition the instructor will answer any questions and will 
identify "trouble spots" for the class. On Friday the group will 
meet with the; instructor and again listen to several repetitions 
of the same tape recordings and discuss the dialogues and the 
enrichment tapes in the context of the history and literature of 
Spanish-speaking countries. This treatment represents typical 
approaches to language Instruction now in use. 

Treatment B. The experiences on Monday of each Week are the 
same for this gr^p as they are for subjects under Treatment A. 

On Friday this group will meet with the Instructor and will listen 
to and view a motion picture film of the dialog studied during the 
week. The films will be repeated the same number of tines as the 
tapes with duscussion organised in the same way as for Treatment A. 
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This trestoeat represents an application of the operant con- 
ditioning model discussed previously, and on the basis of the 
pilot data described above, this treatment is hypothesised to 
result In superior performance (criterion discussed belov) by the 
students. 

Treatment C. The experiences presented to subjects in this 
group will be the same as those for Treatment A except that under 
the treatment subjects will receive the Introductory experience 
on Honday using the film version of the dialogues. This treat- 
ment represents an application of the Gestalt orientation and Is 
similar to that recommended by MLA. 

Treatment 0. Experiences presented to subjects in this 
group will be the same as for those :In Treatment 3 except that 
under this treatment subjects will receive the Introductory 
experience on Monday using the film version of the dialogues. 

essentially the same treatment used by 
Creore and Hai^ell and Is typical of one use of visual materials 
in language Ixi^tructlon. 



Figure 1, The experimental Treatments 



Condltiohs^ 


^tonday 


Tuesday 


Wednesday 


Thursday 


Friday 


A 


Test/tape 


D/JS* 


D/£ 


D/E 


Tape 


B 


Test/tape 


D/E 


D/E 


D/B 


Film 


C 


Test/film 


D/E 


D/E 


D/E 


Tape 


D 


Test/film 


D/E 


D/E 


D/E 


Film 



* D represents individual practice with the dialogue tapes 
E represents Individual practice with the enrichment tapes 











Bvluatlon Procedures , Students will be csvaluated during 
the first portion of each Honday^s class during the quarter* 
Criterion instruments will be administered individually and vill 
consist of verbal responses to a set of standardised stioulus 
estpresslons in Spanish to vhlch the students must respond in 
Spanish. The stiiouli vill be presented by the instructor and 
responses vill be tape recorded for evaluation. Separate test 
stiinili vill be prepared £or each dialogue and vill be based on 
the speech patterns developed in the dialogue. The instructor 
has had extensive experience in rating student responses and has 
developed a numerical scheme for recording and scoring errors 
made in fixed test conversations, student response, granmati- 
cal correctness, and accent, ifumerical scores vill be assigned by 
the instructor. 

Analysis * It is hypothesized that use of the films as intro- 
ductory axparlences vill have a tendency to inhibit the learning 
of the fcrelgn language through the introduction of a negative 
■at reaulting from the Impact of many Irrelevant stimuli. However, 
the use of films after the verbal behavior (depicted in the films) 
has been learned will significantly enhance retention and per- 
formance because a much larger number of stimuli will become 
associated with and will elicit desired responses. Null hypotheses 
of differences between performance scores will be tested using 
analysis of variance techniques. 

While the number of subjects in each cell is relatively 
small (N*15), relatively large amounts of Instruction will be pro- 
vided each subject, making it economically feasible to provide 
such Instruction if the experiment Indicates a preferred instruc- 
tional method. 

Approximate Time Schedule . Development and refinement of 
the measurement instruments, laboratory tapes, and course 
materials will be accooq&lished during the six month period 
from January 1, to June 30, 1964. The experiment td.ll be con- 
ducted during the fall academic quarter. Analysis of the data 
and completion of the final report will require two months 
beginning January 1, 1965. The project will not be active during 
the months of July and August, 1964. 

Expected End Product . The results of this research will 
shed new light on the appropriate use of vieual materials in con- 
junction with foreign language laboratory experiences. It is 
expected that the test materials developed for the project will be 
suitable for use in a variety of Spanish teaching situations. 
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Publication Plans « The results of this research will be 
offered for publication In professional Journ'^ls such as the' 
Audio-Visual Comnunlcatlon Review. 



Appecjdlx B 



Practice Exercise #2 



X. Problem 

In Aiperican education the expectation is virtually universal 
that the adequately trained elementary school teacher will have at 
least some ainimun proficiency in keyboard (piano and organ) 
instrument skills^ Music isstsuctlonj particularly in the pri* ‘ 
mary and intermediate grades, is usually closely integrated in the 
instructional program, and clastroom teachers are expected to be 
relatively self-sufficient in supplyii^ music experiences for 
their students* Consequently, certification requirements and 
degree programs for elementary teachers commonly require either 
specific training or demonstration of proficiency in k^board 
instrument skills* 

This requirement presents music educators in elementary 
teacher education institutions with unique problems* The con- 
tinuous and increasing demands for well-trained elementary school 
teachers, and the increasing number of students choosing this 
profession, exerts pressures for increasing class site and 
student-teacher ratios* However, the nature of classes requir- 
ing perfc-rsas^c dictates that they be small and students have 
opportunity for individual practice and tutoring* Xt is iienemXly 
accepted by rsuslc educators that studemts must receive frequent 
feedback regarding the adequacy of their performance if they are 
to progress ^apidly* Thus we have an enigma with regard to 
efficiency* Small classes end low pupil-teacher ratios are 
inefficient in terms of utilization of staff and facilities, but 
large classes and high pupil^^eacher ratios are inefficient in 
terns of student achievement* 

The question of efficiency can be pursued further* Consider- 
able specialized training and ability are expected of a music 
Instructor, but teaching fundamental keyboard skills to naive 
students dictates that he listen to students perform and tell them 
how they are progressing* Ihich of his time and effort Is dsvoted 
to rudimentary a^ d repetitious instruction, not markedly different 
from that which would be given a young child in private instruc- 
tion* If there is no alternative, the situation can be accepted 
as a x^cessary evil, no matter how regrettable it is that much of 
the Instructor's capabilities are not exercised* But if an alter- 
native exists, then it appears obvious that the feedback-giving 
role is an inefficient use of Instructor's time* 
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The need to acquire keyboard instrument skills presents the 
student with certain problems, too* The elementary education 
teacher candidate who has had no previous musical training is the 
typical case» not the exception. As with dancing, swimming, and 
many other skills demonstrated in nubile oerformance. the most 
acceptable time for learning has passed, and public demonstra- 
tion of ineptitude la humiliating# Yet the nature of the learning 
task requires some active performance, however inept, fr^ him, 
and that he be given feedback regarding the quality of hi's per- 
formance# Private tutoring for all students is simply not 
feasible# Practice without feedback Is Inefficient because 
errors are practiced along with correct perforusnee. And when 
performance and feedback take place In a class setting, the feed- 
back is not merely Informative, but often threatening to the 
student. 

Briefly stated, then, problems to which this proposed pro- 
ject aM Addressed are threefold: 

1) How can more adequate tutorial feedback be pro- 
vided to beginning piano students In eliunentary 
teacher training programs? 

2) How can Instructor time required for rudimentary and 
repetitious phases of piano instruction be reduced? 

3) How can the adverse effects of public practice and 
public feedback on students learsing basic piano 
skills be minimized? 

One stage In the process of acquiring basic piano skills 
where problems mentioned above seem to be concentrated is the 
point where students learn the basic chord patterns in various 
keys. Learning these patterns is essentially a rote process. 

After seeing the correct patterns demonstrated on a piano key- 
board students go through a repetitious process of individual 
praci:ice, performance and appropriate correction by the 
Instructor. 

Oregon College of Education and the Teaching Research 
Division of the Oregon State System of Higher Education are 
presently engaged In a pilot study which Involves the develop- 
ment and testing of an automatic training device to be used In 
teaching this skill, fhe. music staff believes that this 
learning task Is particularly well-suited to aelf-lnstructlonal 
procedures. The educational objective may be classified, 
essentially, as the acquisition of a psychomotor skill. 
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Correct performance is readily distinguished from incorrect; 
either the correct combination of keys is struck, or it is not. 

The striking of the keys is, in a sense, a switching process. 

The piano mechanism translates a finger movement into a musical 
note; a simultaneous combination of movements becomes a correct 
snd pleasing chord, and an Incorrect combination, a discord, 
itodificatlon of a piano to acconq>lish automatic switching throu^ 
electronic means is a relatively simple procedure that need x^t 
Interfere markedly with its more conventional function^ 

Such a device has been constructed. A piano has been modi- 
fied to accomplish switching, as described above. The striking of 
a combination of keys will be converted electronically into input 
for conq[>uter-type logic circuitry capable of discriminating the 
correct pattern from all others and reacting with an output that 
can be translated into meaningful feedback to the student as to 
the correctness of the pattern played. 

Capabilities of this device Include: indication to the 

student of the chord pattern to be played, visual and/or auditory 
feedback to the student of the correctness of his performance at 
each step of a chord progression, recycling to the beginning as 
soon as an error is made, automatic advance to a new chord pro- 
gression as soon as a correct progression has been completed, 
recycling of a given progression for a predetermined nuaaber of 
times (e.g. , six consecutive correct performances before auto- 
matic advance) , or manual control by the student of the chord 
pattern and progression to be practiced* 

The use of this device may free the instructor to a large 
extent from that portion of his instruction that involves repe- 
^Ibiously evaluating and giving feedback on student performance. 
The student may be enabled to practice privately, with continuous, 
accurate, unthreatening automatic feedback regarding the quality 
of his performance. Outcomes of the pilot study now being com- 
pleted Include a pro«.otype of the device, schematic dravings and 
descriptive reports that will enable reproduction of device, 
determination and Improvement of the validity and reliability of 
Instruments to measure performance of the Instructional objec- 
tives and affective reactions to its use, and possible but 
presently unanticipated changes in the design of the device. 

The purpose of the proposed research is to assess the extent to 
which use of this device may alleviate the three-fold problems 
described above. 
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IX • Related ::aadlnas and Research 

The review of the literature for this study is confined to 
two separate areas of research: <1) psychological studies which 
involve the use of feedb&wfc «ud (2) ^sic educational research* 



Feedback . Various studies of knowledge of perfomance, 
(HePherson, Dees and Grindley, 1948-495 Michael and Mtccoby, 1953, 
iamons, 1956) have shown that students Improve in their per- 
formance when they are given knowledge of results* Michael and 
Maccoby concluded that ”the most Important factor in influencing 
the amount of learning In this experiment was the provision of 
l^ledge of correct response (KCR).” Wolfle (1951) stated that 
“laboratory studies are unequivocal in emphasising the liqtor- 
tance of giving a subject s specific and as immediate information 
as possible concerning the outcome of his efforts •“ (p. 1267) 

In 1954 Skinner suggested that laboratory ree^^ch on 
behavior had a direct bearing on the teaching process. He felt 
that application of research findings could be effectively brought 
about by using a mechanical device* This device was to supply 
the student immediately with knowledge of his correct response* 

In 1961 he stated that “exploratory research in schools and 
colleges indicates that what is now taught by teacher, textbook, 
lecture, or film can be tau^t In half the time with half the 
effort by machine 



Music Education Research * The reports of several studies 
of late (Barnes Buchanan, Carleen, Uoelflin) point to the des- 
perate problem faced by music educators in colleges which train 
elementary teachers* Plainly stated, the regular classroom 
teachers are not trained adequately to do an effective job of 
teaching music. Buchanan (1964) stated “The ability to play a 
piano is an asset to a regular classroom teacher, and often times 
it is the determining factor in being selected for the job*.<> 
Something must and can be done to compensate this«».(training)**. 
deficiency. Just as in science, mathematics, languages and other 
areas of learning, music education must streamline and revise its 
methods and procedures of instruction. Certainly this is true of 
piano teaching* The purpose of such a revision is to more effec- 
tively and more economically, from the standpoint of time, prepare 
the prospective teacher to meet his pianlstic needs." 

Carlsen (1962) used programed instruction to develop melodic 
dictation ability and concluded, “The results of the experiment 
clearly Indicated the value of the (method)*.. The potential of 
programed instruction appears great*** to release the tescher for 




tasks which only the teacher can do.'* Woelflln (1963) expert* 
awntad with instrument instruction program machine to free the 
taacher from tasks which amount to supplying feedback. Barnes 
(1963) Introduced programed instruction over factual information 
as such not only "could save many hours of instructor time and 
student time, but the use of the programed book could permit the 
teacher at the outset of the course to assume a specific level of 
coD^etence on the part of c^ery student in the class.*' 

A call for help in the matter of research itself was Issued 
by Fetzold at the Husic Educators National Conference, 1963. He 
comnented that "during the period 1952-62, 70% of the music educa- 
tion dissertations listed in Dissertation Abstracts were the 
relatively uncomplicated and highly popular survey studies and 
30% were basic or^ action type. Often the survey was made in 
desperation and did nothing but help perpetuate the commonplace 
in music education research." He stressed the need of pro- 
fessional team work, l.e. "the teacher must: (1) be relieved of 

a portion of his teaching load in order to have sufficient time to 
carry on investigative activities designed to isqprove the pro- 
gram; (2) be given assistance in planning projects, and consul- 
tative services by trained researchers should be available 
throu^out the course of the projects. Such cooperation. • .will. • . 
result in substantial gains toward improvement of programs and 
instructional procedures." 

III. Objectives 

The proposed research seeks to determine: 

1) Will use of the trainiTing device described elimi- 
nate or significantly reduce the amount of time 
required of the instructor for tutorial or feed- 
back-giving activity? 

2) Will students using the device demonstrate at 
least as much increase in proficiency in playing 
the selected chord patterns on completion as those 
taught in a described conventional manner? 

3) Will students using the device require equal or 
less practice time to reach a criterion of per- 
formance than those taught in a conventio’ial 
manner? 



4) Will studenc attitudes, as measured by a Thurstone 
scale, indicate significantly more positive affect 
toward learning of the skill in question by students 
using the device, than by students receiving 
"conventional" instruction? 

5) If differences in affect are noted, are they due to 
novelty of the device, its effectiveness, or the 
privateness of instruction? 

6) After working with the device, will teachers indicate 
a preference for its continued use, or to return to 
conventional instruction? 



IV. Procedure 



a. General design and selection of sample . 

Two groups of sub^^cts will be used in the study, an experi- 
mentally tau^t group and a conventionally taught group. Subjects 
will be selected from students enrolled in the Ihislc Fundamentals 
classes at Oregon College of Education, Fall term, 1965, and 
Winter term, 1966. Classes of approximately twenty students each 
will be used Fall term, and two other classes of similar size will 
be used Winter term. Since two Instructors teach the course each 
term, they will be rotated in the experiment so that the instruc- 
tor of the experlmeutal group the first term will have the 
conventional group the second. Rotation for time of day will be 
eccoQ^llshed t^imilarly. Subjects will be pretested to determine 
initial level of performance. Prior experience indicates that 
students with initial proficiency are relatively rare, but all 
such students will be eliminated from the experiment. Since 
students in thdse classes are in the same educational program and 
will not be Informed of instructor or treatment until after 
enrollment, it is reasonable to assume that no factors other than 
chance are operating in assignment of individuals to groups, 
with the further expectation that rotation of treatments will 
compensate for any differences in assignment to groups that may 
exist. 



b. E xperimental treatment 

In both experimental and conventional groups the instruc- 
tional objectives will be the playing of fcur-step chord 
progr«>sslon3 in eleven keys, namely G, D, F, B**, A, E, E^, A*’, 
^1» ^ 2 » ^ 3 * with acceptable ten^o and regularity. Subjects will 
be inrormed of the objectives at the outset of the study. 
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All subjects will receive similar instruction In imislc 
fundaneiitals prior to Introduction of the experimental treatment. 
Such instruction will include basic elements of notation atid a 
theoretical explanation of chord building. Sullivan notebooks 
will be used by all subjects. Differences between experimental 
and conventional groups will begin iiimediately following d mma n<^ 

St ration of the basic chord progression. 

From this point to the post-test, subjects in the groups 
using the electronically modified piano will not perform in 
class or receive feedback from the instructor. All practice and 
feedback will be accomplished through use of the device. In the 
regular class meetings for these students, other areas of 
nusicallty, such as music literature, will be presented. 

The conventional group, during this same time, will 
experience an instructional method widely used throughout the 
country* This will consist of each student in turn pciir forming 
certain chord progressions, receiving feedback from the Instructor 
regarding his accuracy, and witnessing the same process enacted 
wirh other students. 

All subjects, experimental and control, will be directed 
to request a post-test when they have completed five consecu- 
tive correct progressions in each of the eleven required keys at 
a single sitting* In the case of the students using the elec- 
tronically modified piano, the device will automatically advance 
when the desired nuxnber of correct repetitions has been accom* 
pushed, so his readiness to take the test will be indicated to 
him automatically. Control group subjects will use a checksheet 
to record their consecutive correct progresslonj to determine 
completion of their training period* The post-test will be 
administered within 36 hours of said completion, and will con- 
sist of the students' actual performance on eleven progressions, 
each of which will be tape recorded* 

c. Data and Instrumentation 

1) Time records. 

a) Instructors of both experimental and conventional 
groups will keep accurate records, of actual time 
spent in a tutorial feedback-giving relation- 
ship with students* While it is expected that 
the instructor of the group using the electroni- 
cally modified piano will spend no time in this 
manner, unanticipated events may necessitate 
some such activity. 
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b) practice time cards will be distributed to all 
students and also will be placed on all school 
pianos. Subjects will record all practice time, 
whether spent on school or home pianos or in 
performance in class = Total time spent by 
each student will be confuted. 

2) Post-test scores. 

The tape recorded post-tests will be evaluated by three 
con^etent music instructors, who will not know the iden- 
tity of the subject whose test is being evaluated, or 
whether he was in the experimental or conventional group. 
Indices of interrater reliability will be determined and 
reported. The score assigned to each subject's performance 
will be the average of the three scores given by the 
evaluators. 

Post-test scores will be weighted 60% on accuracy, 

25% on tenq)o, and 15% on regularity (steadiness) . The 
accuracy score will be determined by scoring each of the 
four steps in each of the eleven progressions as either 
correct or incorrect. Tempo will be scored on a five- 
point scale, ranging from one point for less than 40 
chords per minute to five points for 96 or more chords 
per minute ; Steadiness, or regularity, refers to the 
fluidity or lack of hesitation demonstrated in playing 
the chords. This also will be rated on a five-point 
scale. 

3) Thurstone scale. 

A Thurstone-type scale will be constructed to meastire 
the affective reactions of the experimental and conven- 
tional group subjects to the type of instruction in 
chording that they received. In addition, anonymous, 
open-ended written statements will be collected from the 
8tud&9tts, to discover unanticipated student responses 
to the treatments. 

4) Instructor reactions. 

Since the members of the music department have already 
expressed enthusiastic support of the project, it does not 
appear that a systematic comparison of their reactions to 
the respective treatments would be appropriate. However, 
a narrative report on their experiences will be made, in 
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which they will indicate whether, after naing the 
training device, they wo^ld prefer to continue using 
it or to return to conventional instruction, how they 
would like to see the device or its application modi* 
fied, and whether they would like to see an extension 
of this training principle Into other areas of ^helr 
course. 

d. Analysis of the data 

Significance of the difference between the experimental and 
control groups in instructor feedback time, student practice time, 
student p 08 t*test scores, and affective reactions of students to 
the respective treatments, will be determined by t*tests. 

e. Approximate time schedule 

July 1 - September 30, 1965 - Modification of two addi- 
tional pianos; construc- 
tion of measurement Instru- 
ments; orientation of staff. 

October 1, 1965 - March 31, 1966 - Experimental treatment; 

data collection. 

April 1 - June 30, 1966 - Data processing; writing of 

final report. 

Outcomes of the proposed research effort will include: 

1) A research report providing evidence on the questions 
asked above. 

2) The existence of three use-tested devices for instruc- 
tion in chordlng skills ^ available for demonstration 
to music educators and research specialists. 

3) Schematic diagrams and descriptive literature to 
allow replication of the training device. 

4) Information regarding possible modification of the 
device or Its utilization. 
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Appendix C 



Class Exercise #1 



Procedure 

General Method and Research Strategy 

The general plan of research Is to compare changes In per- 
formance of five, randomly assigned groups of Inservlce teachers 
on criterion measures developed by Schalock, Beaird, Simmons, 

(see Appendix) for recording teacher behavior* The five groups 
represent the presence or absence of three conditions: (1) a 

workshop experience dealing with observation, self-evaluation 
and goal setting; (2) self-observation via video recording; and 
(3) overt self-evaluation via video-recording. The design permits 
the Investigation of effects of self-evaluatlonal procedures using 
video recording, workshop experience, and overt self-evaluation 
procedures on behavioral change. 

Sample Plan . Sixty volunteer teachers from the Corvallis 
Public School System will take part in this study. During the 
first year (1966-67) thirty teachers will participate with 
twenty teachers in Group X and ten teachers In Group II (see 
page 12). The second year (1967-1968) thirty teachers ^11 par- 
ticipate as follows: ten In Group III, ten In Group IV, and ten 

In Group V (see page 12). 

The use of volunteers In this study deserves consideration 
at this point because important differences may exist between 
teachers who volunteer for self-in^rovement experiences, such as 
those provided by this study, and those who do not volunteer or 
must be directed to take part by some superior authority. 

Following the pilot project (outlined In Related Research 
above) all teachers who had seen the demonstrations were willing 
to take part In a further study of self-evaluation via video 
recording. This willingness to volunteer Is considered to be a 
major factor In the potential success of this study. Because the 
Initiative for this study comes from the Professional Improvement 
Coxnolttoe, Corvallis Education Association of the Corvallis Public 
Schools t i *^*9 the teachers themselves, (the Principal 
Investigator Is Chairman of this committee) the variance between 

volunteer teachers and those required to take part In such a study 
cannot be examined legitimately. If self-evaluation through 
video recording Is found to be a successful method of changing 
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teacher behavior* further studies will undoubtedly examine this 
variable. 

Even though recording procedures developed by Schalock, et 

VWWV 'mm M*^WM V «»«.«*%»%» •» w •» ^ A. «4V»WO ^ MV»WOa«CV«* «»a/U4W 

room situation both emphasizes the Interpersonal communication 
processes observed and quantified and provides a more advan-* 
tageotis recording situation* volunteer teachers in the third 
through eighth grades will be used in thl saoq>le. If suffi- 
cient teachers volxinteer* this range in g&ade level will be 
collapsed downward to provide as much I ^mogenelty on this 
variable as possible. Specifically* the Corvallis p d>lic schools 
employ approximately 25 teachers at each elementary aid junior 
high school grade level. Of these 25 teachers at each grade 
level approximately 20 have had more than three years* teaching 
experience* but fev/er than 10 years* experience. Since 60 
teachers will participate in the study on a voluntary basis* it 
will be necessary to sample from a minimum of three grade levels* 
Attempts will be made to restrict the sample to third, fourth, 
and fifth grade teachers with the bulk of the sample being drawn 
from the fourth g:i:ade and the other two grades being sampled 
equally. 

Treatment g nd procedure . The five groups of teachers 
taking part in this study will experience one or more of the 
following conditi.ons: 

1. Condition W ( Workshop) . A two-day workshop including 
training in video recording observation, using pilot project films 
and classroom simulation films (Kersh* 1963); evaluation pro- 
cedures* and the setting of behavioral objectives. Emphasis 
throughout the workshop will be on the development of personal, 
overt standards of evaluation with a view toward self-evaluation. 
This workshop will be conducted by the Principal Investigator, 
Research Director* and resource staff from the Teaching Research 
Division of the Oregon State System of Higher Education. The 
workshop will use observational procedures developed by Schalock, 
Beaird, and Sinsnons (1964), classroom simulation films and 
techniques developed by Kersh (1963) , and a slide tape presenta- 
tion designed to train persons to identify and state behavioral 
objectives developed by Paulson (1965) . All the above named 
researchers are currently in the Teaching Research Division and 
can be used as resource persons at the workshop. The workshop 
will be pres«;>ated at the beginning of each year of the study. 






2» Condition SO ( Self-Observation) • The viewing of a 
playback of the video recording by the teacher, alone In the 
room, for the purpose of self^observation 

3. Condition SE ( Overt Self-Evaluation) • Although it can 
be assumed chat some sort of self-evaluative procedures are 
followed by all individuals confronted with a record of their 
own performance, whether or not the teacher writes down his 
criticisms of himself and makes written statements of his 
behavioral goals In the form demonstrated at the workshop is a 
controllable factor and differentiates this condition from con- 
dition SO* 

Treatment Groups * During each of the two years of the 
study, thirty teachers will be rai^omly assigned to treatm^t 
groups to experience one or more of the conditions listed above. 
During the first year, two of the five treatment groups will be 
used to test the major objective which isi; self-observation and 
self-evaluation will result in an over-all improvement of teacher 
effectiveness as defined aifd evidenced by her management 
behaviors and Interpersonal orientation* 

Group One * Twenty teachers will attend the workshop and 
participate in the self-observation arid self-evaluation pro- 
cedures. Because this group will experience all experimental con- 
ditions (workshop, self-observation, at^ self-evaluation), it is 
desirable to expose as many to the treatment as possible; there- 
fore twice as many subjects will be utilized In this group as in 
other experimental and control groups* This group will have 
three recordings of their classroom behavior made and will view 
the first two* All three recordings will form data for analysis 
(W:SO:SE, N-20). 

Group Two * A control group of ten teachers will attend the 
workshop and have pre- and post-experiment video recordings made 
of their classroom behavior. They will not participate in the 
self-observation or self-evaluation procedures (W:no-So:no-SE, 
N-10) * 



During the second year three alternate conditions (as listed 
below) of the experiment ai group will be tested in an attempt to 
evaluate and differentiate the effects of the workshop and overt 
evaluation processes* Groups three, four, and five %dll all be 
recorded three times and will observe their own behavior after 
the first two recordings* All three recordings will form data 
for analysis* 
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CgQUP Three # Ten teachers will attend the second workshop 
and participate in the self-observation process; they will not go 
through the overt self-evaluation procedures (W: SO: no-SB, N»10)« 

Girotati Pout* Tah trill 1*1^^ 

will participate in the self-observation and self-evaluation 
processes using the forms provided for self-criticism and overt 
goal setting as best they can (no-W:S0:SE, N»10)# 

Orottp Five # Ten teachers will not attend the workshop, will 
participate in the self-observation process and will not go 
through the self-evaluation procedures (no-W:SO:no-SE, N«10), 
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Evaluation Sheet 



We need to know how well the ideas and Issues in this manual are 
communicated to you* You arc the test audience for this material* 
To remove or strengthen the weak spots » to retain or iprove the 
strong ones (if any), an account of your learning experience 
vou read these sections is crucial. Use this sheet as you study 

and jot down your reactions, uvu u uc 

errors. We* re concerned with how the message is coming through* 



SECTION# TITLE 



Give topic. 


Importance 


Clarity 


Suggested 


paragraph or 


Rate 


Rate 


Improvements 


sentence, page 


1 2 3 4 5 


1 2 3 4 5 






Low - High 


Low - High 








Over-cll Rating of 
Important 


the Section: 


Use f» 5-point Scale 

Understandable _ 




12 3 


4 5 


12 3 4 


5 


Very 


Very 


Very 


Very 


Low 


High 


Low 


High 


(Circle one) 




(Circle one) 
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SECTION VII 
Data Analysis I 
James H. Bealrd 



In detemlnlikg the appropriate analysis technique one must 
consider the following factors: 

1. I<Ihat is my research question? 

2. IThat is the nature of my data? 

3. How many samples do I have? 

A. Are my samples related or Independent? 

Three types of research questions will be considered: (1) 

Experimental or Comparative, (2) Relational, and (3) Descriptive. 

For each type of question one of several techniques may be used 
for analysis depending on other factors, e.g. the level of measure* 
ment available, the number of samples, the independence of the samples 
Let's consider each of these In turn. 

Levels of Measurement 

Four levels of measurement may be Identified. These levels 
are defined in terms of the scaling characteristics of quantifi- 
able descriptors (numbers) each provides. The four levels are; 
nominal, ordinal. Interval and ratio. 

Nominal Scales . Nominal scales are those in which the numbers 
are merely substitute names for the objects being scaled. In the 
usual quantitative sense the number has no meaning. That is, the 
number does not indicate that the object has more or less of a 
characteristic than does any other object. Probably the most com- 
mon use of a nominal scale are the numbers assigned to various 
members of an athletic team. Another commonly used system of nom- 
inal numbers is the Dewey Decimal System employed by many libraries. 

List one or two other such systems: 

1 . 



2 . 



Note: Compare your answers \7ith those in the answer box on the 

next page. 
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Answers : Nominal Scales 

jX, Sex (o » boys, 1 - girls) 

(2, College Departments (1 « Anthropology, 2 » 

Biology, 3 *» Chemistry, etc.) 

3. Makes of cars (1 * Chevrolets, 2 « Ford, 

3-* Dodge, etiy.) 

Ordinal Scales . The lowest level of scales in which numbers 
have quantitative meaning are ordinal scales. As In nominal scales 
these numbers also differentiate between the objects being scaled; 
however, in addition the numbers Indicate xvhlch of two objects 
have more of the characterlettc. In some states, for example, 
schools are classified according to siz^ of enrollment. The num- 
bers assigned to the classes indicate a difference In sis*e, e,g. 
Class 1 schools are longer than Class II schools, etc. If I were 
to order the Institute members according to their height, assign- 
ing the number 1 to the tallest, 2 to the second tallest, etc,, 
the set of numbers would be an ordinal scale. 

Hhftt are some ordinal scales that yon have experienced? 

1 , 

2 . 

3. 

Note: Compare your answers with those in the answer box on 

the next page* 
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Answers : Cz ^Inal Scales 



I I. Order preference for a list of foods 
2. List of occupations ordered by amount of 
Income • 

! 3. Course grades (usually) | 

Interval Scales . Interval scales possess all of the charac- 
teristics of nominal and ordinal scales, l.e«, they use numbers 
to name objects and the numbers have quantitative meaning, e.g. a 
walue of 10 may mean that the object has more of the character- 
istic than does the object labled 9, or 8, or 3. One important ad- 
ditioaml characteristic is added to the interval scale and that 
is that consecutive numbers are equally spaced along the scale, 
i.e« the intervals between the numbers are equal. This means 
that the distance between 9 and 13, for example, is equal to the 
distance between 7 or 3. The Fahrenheit thermometer is an inter- 
val scale. Is the centigrade thermometer ? On an interval scale 
is 8 twice as large as 4? ^y? 

List two examples of interval sclaes. 

1 . 



2 . 

Not^* Compare your answers with those in the answer box on the 
next page. 
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Answers: Interval Scales 

j 1. Serial numbers assigned to equidistant teie-| 
I phone poles s 

j 2» Heights of airplanes above a eiven clfev. 

I “ ' " 



X 

Mtip Scales. The highest order of scales Is the ratio scale 
which adds a "zero point" to the intetval scale. This addition is 
Important since it permits us to say f:hat one score is many times 
greater than another » e*g. 16 is tv^ce as large as 8 or 4 times 
greater than A or 1/3 as large as 48 • Measures of distance or 
weight are ratio scales. 



The following depict the four types of scales, label each. 



1 . 

2 . 

3. 

4. 
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Mewers: Types of scales 



1. 


Ratio 


2. 


NomlXf ox 


1 3. 


Ordinal 


L" 


Interval 



Most; loeasures of psychological attributes fall to meet the 
criteria of ratio scales. In fact they rarely are sophisticated 
enough to be interval scales. They often are more than ordinal. 
Therefore, the data level will hereafter be referred to as being: 

Class C: homlnal only 

Class B: Ordinal only 

Class A: Better than ordinal 



Number of Samples 

Comparative studies may be made using twc or more saxples. 
The distinction must be made between situations in which just two 
samples are being compared and In 'tdiich k samples (any number of 
' ' 4 >les) are being compared. 

Let*8 consider two sample studies first. Anytime subjects 
of one type are compared with subjects of another type, a two 
sample comparison is involved. This Implies to the Invcistlgator 
that he must choose one group to represent one type of subjects 
and another group to represent the second type of subjects. 
Comparisons might also be made of subjects training under our 
set of conditions with these who trained under another set ox 

conditions. 



1 . 

2 . 



Men 


"i 

1 vs. 

>( 


Women | 


Sighted 


vs. 


Blind 



Mien more then two samples are Involved several conditions 
must be considered* Suppose in a given company we wish to compare 
attitudes towards mlitaiit Civil Kights activities held by (1) 
Management personnel, (2) supervisory personnel, and (3) nonsuper- 
vlsory personnel* Obviously we would be required to select three 
samples (one for each classification of personnel)* A similar 
situation might be the comparison of effectiveness of three 
teaching techniques* 



In each case the categorizations was made along a single 
factor or dimension, e*g. type of worker, and teaching method* 
Graphically such studies would be portrayed as follows: 



1* Categories in one dimension 



Teaching Method 




B i C j 



School 



b* 



r 

j 

&iglneerlng | Education 



Arts & Sciences 



Suppose in the second example, we suspected different learning 
capabilities for boys and girls. We may then add a second dimen-* 
Sion, l.e* a dimension for sex which has two categories. Addition 
of this dimension increases the number of samples required. In 
this example we have three categories of method and two categories 
of sex, and must, therefore have six samples (3X2). 

There is theoretically, no limit to the number of dimensions 
which might be included in a study. The point to be made here is 
that the number of samples Increases by a multiple of . • number 
of categories in each added dimension. 



Graphically^ studies in two or snore dimensions are portrayed 
as follows : 



8 • 



Teaching Method 




Large 

Size of Class 



Small 



Independence of Samples 

Samples fall Into one of two categories-*-they are either re- 
lated or are independent * Indenendent samolea are these which 
were randomly or pseudo-randomly selected# Ihls means that aa}^ 
case within a population of cases had an equal opportunity to be 
selected# For most studies utilising parametric statistical pro- 
cedures, it is assumed that the samples were randomly selected 
from infinitely large, normally distributed populations# 

Technically spealclng, it Is difficult to justify the random- 
ness of samples used in many educational experiments# Psycholog- 
ists, for example, often will assign various sections of a course 
they are teaching to different treatment groups and conduct their 
study as though each section was a random sample draxm from some 
hypothetical infinitely large populations. I am certain that 
this practice stimulates all sorts of anxiety, wrath, and scorn 
in the statistical purists* Such practice, however contaminated, 
seldom results in deeislcns different from those x?hich might re- 
sult from using completely random samples# In my opinion, the 
three assumptlorzs (Randomness * Infinite populations, normally dis- 
tributed) can be horrendously violated without affecting the de- 
cision to be made* 

Related samples are those which do not, for various reasons, 
meet criteria of randomness. An example of related samples is the 
comparison of pre* and post-training scores for a group of subjects* 
Obviously post training performance can be expected to be related 
to Initial performance. Similarly if all subjects in a study are 
subjected to all treatments, the samples are related. Analysis 
techniques differ for studies using related and independent samples. 

A Note on Hypotheses 

For studies making comparisons our analysis techniques per- 
mit only for testing of hypotheses of equality, e.g.# the differences 
between treatment groups is equal to zero (the null), the corre- 
lation between two sets of scores is-1.00. T7e then provide evi- 
dence which permits or fails to permit us to reject such a hy- 
pothesis • 

I would like to focus these comments briefly on three formal 
null hypotheses. 

Very frequently, research involves the comparison of sample 
means and variances* Far more often than not, the researcher 
conceives the scores of the two (or more) samples as possible 



members of a single population of scores. The scores may come 
from persons, objects, or ©vents that could themselves hardly be 
considered as a random sample from a single population. There 
^ould be a sample of boys' scores and a sample of girls' scores. 
Tne boys could hardly be considered a random aample from a mixed 
Population, nor could the girls. But, if the scores are unre- 
lated to the sex characteristic, (and with the null hypothesis, 
that vould be hypothesized), then a single population of scores 
is a tenable hypothesis. The full null hypothesis for this gen- 
eral case is: 

Hull ^rpothesis I: The differences between che means and 
variances of the samples are no greater than differences due 
to the vagaries of random aanpling from a single, homally 
dictributed, infinite populatioix. 

Two or more samples coi be analyzed at the same time with this 
hypothesis. 

If a test of two sample variances is to be made, with no con- 
cern for the means, the null hypothesis is: 

Hull Hypothesis II; The difference between the variances of 
the samples is no greater than the differences to be expected 
from the vagaries of random sampling from a single, normally 
distributed, infinite population. 

Suppose that the second null hypothesis Is tested and re- 
jected. The only proper conclusion is that the samples do not 
represent a single population. Sometimes texts se^ to imply 
that there could be a single normally distributed population that 
yields random samples with heterogeneous variances. This author 
believes that they should not so imply. To reject the second null 
hypothesis is to reject the applicability of the first, and the 
first should not be tested after the second is rejected. 

If, oti the other hand, the second hypothesis is tested end 
not rejected, that Is to say, the variances are not different, 
then the first hypothesis can be tested. If the first hypothesis 
is then rejected, the differences between samples can be attributed 
mostly to the difference between sample means. The difference 
between variances still contributes a little to the difference be- 
tween samples, but it was demonstrated with the second hypothesis 
that the difference between variances was not sufficiently great 
to be called statistically significant. Rigorously speaking, it 
is not proper to speak of a difference In means based upon a test 
of the first hypothesis. In practice, however, such a difference 
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Is cXalmed if It has been demonstrated that the difference be- 
tween sample variances Is not statistically significant. 

If the variances of samples are found to be different by 
testing the second hypothesis, there is a third hypothesis that 
can be used to compare the sample means for two samples. This 
hypothesis is based on the assumption that there are parent 
populations with equal means. It makes no difference whether 
the variances of these populations (or their samples) are the 
same or different. This hypothesis Iss 

Null Hypothesis III: The difference betvxeen means of the 

two samples is no greater than the differences to be expected 
from the vagaries of random sampling from two normally dist- 
ributed, infinite populations with equal means. 

It seems to this author that this third hypothesis could be 
used far more than it is. THien the two samples have been selected 
because they represent two different groups, such as boys and girls, 
this hypothesis seems to be most appropriate. It has already been 
pointed out that the scores from these different groups can be 
thought of as representing a single population of scores, even 
though neither the boys nor girls can represent a mixed population 
of children. (This type of study in which two groups that differ 
in some specified way are compared as to another attribute is 
known as a causal-comparative study.) It seems, then, that either 
Hypothesis I or III can be properly used co compare means in a 
causal-comparative study. 

Causal-comparative and Ibcperimental Studies . One thing about 
looking at this problem with the author’s bias is the nicety with 
vdiich the statistical methods for causal-comparative and experi- 
mental studies are distinguished. In the experimental study, two 
groups, essentially identical at the outset, are compared as to 
some attribute after different experimental treatments have been 
administered. At the outset of the experiment, it is evident 
that the research worker feels that the samples represent a sin- 
gle population. He wonders if, after experimental treatment, they 
still represent a single population, or if the treatment has caused 
a difference. Therefore, he tests Null Hypothesis I. Thus, as 
the author sees it, Null Hypothesis I is appropriate for experi- 
mental studies and Hypothesis III is preferable for causal-com- 
parative studies. 
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Here are two examples. From an apparently homogeneous com- 
munity of v/hlte rats two groups are randomly selected. One group 
learns that a right turn on the maze, regardless of Its orientat- 
ion, leads to reward. The second group learns that a turn toward 
the window, regardless of the orientation of the maze, leads to 
reward. The learning scores of these two groups should be com- 
pared bv testinn Null Rvnothesi a T ^ Tilovi* . A 

- V ' IT — - - -w-.- y WO 

learns to turn toward the window to earn the reward. The "toward 

window" learning scores of the hooded rats are compared to those 
of the white rats by testing either Null Itypothesls I or III. If 
the researcher chooses to emphasize the possible, overall homo- 
geneity of the learning scores Null ^othesls I la better. |f 
he wishes to emphasize the fact that white rats and hooded rats 
are different In some respects Null Hypothesis III is better. Ip 
either case, the decision to use Hypothesis I or HI should be 
based on his conceptions of the populations that are represented, 
not on characteristics of the samples.. Hypothesis I states that 
there is a single ]i>opulatlon and experimental studies start with 
a single population. Hypothesis HI states that there are two 
populations and causal comparative studies dlstln ulsh betwppp 

two or more samples on an a priori basis. What could be nicer 
than that? 

Degrees freedom . The confidence that can be placed In 
conclusl<xi8 Is related to the 'degrees or freedom" associated with 
the analysis. The concept "degrees of freedom" Is quite complex 
and will not be covered In any depth here. Suffice it to say that 
usually the degrees of freedom Is one less than thu number of olser- 
vatlons, l.e. if you have 20 cases in your sample (n«20) the 
degrees of freedom Is 19 (d.f.«n-l or In this esse d.f .»20-l«19). 
Maybe an cszample will help to clarify this. 

Suppose we have a large bowl containing "peas", each of which 
has a number between 1 and 10 printed on It. Lets say I draw two 
peas (n«2> such that their sum equals 10. with the limiting factor, 
their sum equal 10, I have only one opportunity to select a 

pea randomly. That Is to say I have lost one degree of freedom 
or one opportunity for random selection. For example If I randomly 
select a pea with a 3 on it, I cannot randomly dip In to get my 
second pea for it must equal 7, l.e., (10-3). 

It Is sometimes pointed out that Hypothesis I Is evaluated 
with more degrees of freedom than Hypothesis III, and therefore 
is preferable. That Is getting the cart before the horse. It Is 
true that more degrees of freedom are used with Hypothesis I. I^en 
It is assumed that each case comes from a single population every 




VII-11 



case can be used to estimate tbe population paramenters. ^en it 
is assumed that each sample comes from a separate population only 
the cases within a sample can used to estimate its population 
parameters* Since degrees of freedom are closely related to the 
ntpiber of cases wlilch can be used for estimating paraaeterst 
Hypothesis I does indeed have a greater numb«>r of degrees of free- 
dom. This Is not justification, however, for using Ifypothesls I, 
when it Is not reasonable to assume that the samples represent a 
single population. 

Unstated assumptions . As these three formal hypotheses have 
been stated there are no assumptions left unspecified. If the null 
hypotheses are abbreviated to "There is no difference between 
samples,: as sometimes is the case, the reader cannot be certain 
what the proper statistical test should he. If the reader must 
guess, it Is best to guess that the research worker was assuming 
that random samples have been drawn from a single, normally di8trl~ 
buted. Infinite population. The four assumptions of the previous 
statement (random selection, single population, normally dlstrl** 
buted population. Infinite population) are the most common assump- 
tions for testing hypotheses. 
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Evaluation Sheet 



We need to know how well the ideas and issues in this manual are 
comnainicated to you. You are the test audience for this material. 
To remove or strengthen the weak spots » to retain or improve the 
strong ones (if any) , an account of your learning experience 
you read these sections is crucial. Use this sheet as you study 
and jot down your reactions. Don't be concerned with t^rpo graphical 
errors. We* re concerned with how the message is coming through. 

SECTION _i TITLE 



Giv*’ topic, Itoportance Clarity Suggested 

paragraph or Rate Rate . Improvements 

sentence, page 123 45 12345 

Low “ High Low -High 



Over-all Rating of the Section: Use a 5-point Scale 



Important Understandable 



1 


2 3 


4 5 


1 


2 3 4 


5 


Very 

Low 


(Circle one) 


Very 

High 


Very 

Low 


(Circle one) 


Very 

High 
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SECTION VIII 



Data Analysis II 
Janes H* Deaird 



In the previous Data Analysis section ve considered questions 
which must be asked In order Co select appropriate analysis tools 
and defined the parameters necessary to answer the questions* 

This section will "program” you through the quest wOns and Identify 
for you the "tools” required. Bach tool Is then described. 

The "Program” 

Selection of appropriate data analysis tools is a natter of 
asking (and answering) a set of relevant questions.. This section 
Is Intended to permit you to identify the appropriate analysis 
technique for various types of research problems. 

Begin by asking and answering the question^ "Do I want to 
compare » relate 9 or describe?” 

Compare - turn to A1 on page VIlI-1 

Relate - turn to B 1 on page VIII -1 

Describe turn to Cl on page VIII -1 



You said you want to compare. Now ask the question^ "How 
many groups (samples)?" 

Two samples - turn to A2 on page VIII-2, 
s samples (more than two) - turn to B2 on page VIII-2. 




You said you want to relate. Now ask the quest lon» "How many 
variables?” 

Two variables = turn to A9 on page VIII- 
K variables - turn to B9 on page VIII- 




You said you want to describe. Nov ask the question, "Within 
which measurement level do my data fall?" 



Class A —turn to Ai2 on page VIII- 
Class B - turn to B12 on page VIII- 
Class C - turn to C12 on page VIII- 




You said there were two samples* 
or Independent?" 



"ilre the samples related 



Related turn to A3 on page VIIX-2. 
Independent * turn to B3 on page VI1I*2. 




You said there were k samples, 
ment?" 



"What Is the level of measure- 



Class A - turn to A6 on page VI1I*3, 
Class B - turn to B6 on page VIII-3. 
Class C - turn to C6 on page VIlI-3. 




A3j You 



said related. 



"What is the level of measurement?" 



Class A - turn to A4 on page VIII-2. 
Class B - turn to BA on page VIXI-2. 
Class C - turn to C4 on page VIII-2. 




B3) You 



said Independent. 



"What Is the level of measurement?" 



Class A - turn to A5 on page VIII-3. 
Class B - turn to B5 on page VIII-3. 
Class C - turn to C5 on page VIII-3 » 




You said you are comparing vo related samples with Class A 
data! Use the t-test for correlated samples. 




You said you are comparing tko related samples with Class B 
data! Use the Wllcoxon matched-pairs signed-ranks test 
(see Siegel, 1958, pp.. ?5-83). 



You said you are comparing two related samples with Class C 
data! Use the McNemar test for significance of changes 
(see Siegel, 1956, pp. 53-67). 
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You said you are comparing two independent samples with 
Class A data! Use the t-test for separate group or pooled 
varisnee (recall discussion of null hypotheses). 



You said you are comparing two independent samples with 
Class B data! Use the Mann-Whitney U-test (see Siegel* 



pp. 116-127) 



1956 



You said you are comparing two independent samples with 
Class C data! Use the Chi-aquare test. 




6 ] 



You eaid you had k sam]>le3 with Class A data! 
of variitnee (ABCVA). 



Use analysis 



You said Class B data. 
Indepen^ient?" 



*'Are your samples related or 



Related - turn to A7 on page YIII-3. 
Independent - turn to B7 on page VIII-3. 




You said Class C data, 
independent? 



**Are your samples related or 

I 



Related - turn to A8 on page VIII-4. 
Independent - turn to B8 on page VIXI-4. 




You said you are comparing ic related sasiples using Class B 
data. Use the Friedman two-way analysis of variance (see 
(see Siegel* 1956, pp. 166-172). 



V 

You said you are comparing tc independent samples using 
Class B date. Use the Kruskal-Wallls one-way analysis of 
variance (see Siegel, 1955, pp. 184-193). 
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You said you are comparltis k related sss^les uslag Class C 
data! Use the Cochran Q-test (see Siegel, 1956, pp. 161-166> 

You said you are comparing k independent samples using Clans 
C data* lisa the Chl<-square test« 



You said you want to relate t^«o variables* 
measurement are the variables?" 



”What level of 



Glass A - turn to AlO on page VllI-4. 

Claes B - turn to BIO on page VIII-A* 

Class C - turn to CIO on page VIII-A* 

They are mixed turn to DIO on page VlIX-4* 



@ You said you want to relate ic variables > ”What level of 
measurement is the dependent variable? 

Class A - turn to All on page VIII-5* 

Class B - turn to Bll on page vm-5. 

Class C - turn to Cll on page VIII-5* 




You said you want to relate two Class A variables! Use the 
Pearson product-moment coefficient of correlation (r)* 




You said you want to relate two Class B variables! Use the 
Speanoan rank-order coefficient of correlation (rho) or the 
Kendall rank correlation (Tau)» 



You said you want to relate two Class C variables! Use the 
contingency coefficient (C). 




You said your variables are mixed! If one va:i‘iabXe is 
Class A and the other Class C use biserlal correlation* 
If one is Class A and the other Class B, convert the 
Class A data to Class B and use the Spearman rho or 
Kendall Tau. 
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MU You said you want to relate )c variables where the dependent 
variable is Class Jkl Use multiple regression analysis* 



wlU You said you want to relate ic variables Where toe dependent 
variable is Class B! Use the Kendall partial rank- 
correlation (see Siegel » 1956, pp. 223-229). 




You said you want to relate k variables Where the dependent 
variable Claes Cl Use discriminant analysis • 




You want to describe Clase A data I 
or node end variance (o^). 



Use the mean (X), median 




You want to describe Claes B data! 
You want to describe Class C data! 



Use the median. 
Use the node. 



The "Tools** 

Analysis of Variance . The most powerful single tool for 
experimental studies where there is a single dependent variable 
Is the analysis of variance model (AHOVA) • The model is appli- 
cable for all data that would fall within the Class A level of 
measurement. It may be used anytime that assumptioiita of inde- 
pendent samples drawn from noxisally distributed populations are 
met. When comparlx^ twc Independen:: samples ANOVA la equivalent 
to the t-teat. 



t-Tests . When data fall within the Class A Ic^el, differ- 
ences between two samples may be tested using a t-test. The 
most appropriate t-test for most situations is the t-tost for 
separate group variance. This is especially true for causal- 
comparative studies. For experimental studies, pooled variance 
t-tests are usually more appropriate although the distraction 
becomes sticky and requires close attention to some specific 
details. Let's consider a couple of examples c 

Let's say that ve have devised a new technique for teaching 
reading that will lead to greater retention of facts. We have 
a "good" test for measuring retention that yields data in Class 
A. To determine whether or not our new reading instruction is 



superior to what we have been doing, we randonly assign our class 
to two groups, each of vjhich will receive instruction under one 
of the techniques* The appropriate t*test is for pooled variance* 
The reason 1^ that I^Anrlni^V /cst^nri'i 

■ - - - — — ^ I. w y rw«w \ 

from a single population and have then added an exporlmental 
variable. We are trying to determine if the two ssniples could 
be e3q)ected to have been drawn from a single population. - 

In another study let’s say that we wish to determine w|iether 
sex Is a possible determlnent of quantitative skills in children. 
Randomly selecting a sample of boys and a sample of girls frpm 
a population of tenth-graders (age 15) we again use a t-test, but 
this time the t-test for separate group variance is appropriate. 
Reasonably, we are sampling from two separate populations a n d 
are assuming that the population means are equal. 

In another study we wish to learn whether any change in 
creativity results from training in problem solving. We design 
our study by measuring pretraining levels of creative ability In 
our sample, giving the group training, and then post-testing 
their behevlor on t^Ite same or a demonstrated equivalent measure* 
Here it is obvious that some relationship between pre- and post- 
training performance would be expected » The t— tes»; for correlated 
samples is appropriate. 

Chi-Square . When data fall within Class C, the Chl-equare 
is appropriate for testing differences between two or more 
independent samples. Essentially the Chi-square test Is a test 

of independence as well as difference* 

\ 

An investigator xd.shed to determine whether college graduates 
were more likely to support school levies than were noncollege 
graduates. His findings are recorded in the following contingency 
table s 





Grade 


Non Grade 




For 


75 


148 


223 


Against 


32 


ill 


143 




107 

\ 

4 


259 


366 
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The obtained Chi-square value of 3*70 was not sufficiently large 
to reject the hypothesis that voting behavior was independent of 
graduate status. 

In another study the investigator devised five alternative 
uses for a set of naterials, designed to teach a given behavior. 
After training he noted the following distribution of those who 
did and did not exhibit the behavior. 



Method 





A 


B 


C 


D 


E 




Did 


16 


8 


8 


19 


21 


72 


Did Not 


7 


6 


13 


7 


10 


43 




23 


14 


21 


26 


31 


115 



Again the obtained Chi-square valne of 7.58 was insufficiently 
large to suggest that acquisition of the behavior was better 
for any given method. 

The investigator then wished to know how the trainee's 
attitudes toward the methods differed. He asked them whether 
they were positive, negative or neutral in their attitudes 
toward the method and found the following distribution: 





A 


B 


C 


D 


E 


Total 


Positive 


8 


5 


10 


13 


18 


54 


Negative 


5 


6 


5 


6 


8 


30 


Neutral 


10 


3 


6 


7 


5 


31 


Total 


23 


14 


21 


26 


31 


115 



The obtained Chi-square value was 7.50, again sho^wing attitudes 
Independent of method. 



VIII-7 






fa 



tmsi 



The Uniting factor of Chl- aouare is that oaly two dlnen- 
siona may be categorized* Indeed this limitation holdj for all 
nonparametric techniques* 

Studies jf Relationship 

Single Order Correlation * tlhen an lovestlgatox wishes to 
determine the <octent of relationship between two characteristics » 
measures of both of which fall within the Class A level, the 
Pearson product-nneasurement coefficient of correlation (r) is 
the appropriate tool in most cases. The most crucial, limiting 
factor, which might alter the use of this approach is the 
collnearity of the two variables* If the relationship of 
variables has been studied previously and evidence of linearity 
is present, use of r is generally safe* Without such evidence 
the careful investigator should first produce a scatteirplot of 
his data, studying the plot to determine any obvious deviations 
from linearity* If the scatterplot provides rather cleiir evi- 
dence of curvilinear ity, alternative correlational techtiques 
should be used (Wert, et al*, 1956)* 

An investigator wished to determine the relationship between 
quantitative aptitudes of mathematics Instructors and achievement 
of their students* His first analysis of this relationship indi- 
cated little relationship (r • *16)* The scatterplot for his 
data is shown below: 



Class 

Achievement 









A 

T 



A. 

T 



Instructor Aptitude 






A 

T 



Wbuld he be jiustifled in concluding that little relationsmip 
was present? 
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Another Imres tlgat or » in reviewing creativity research, 
found the statement that a rather clear relationship between 
creative behavior existed for those with l.Q. *s of 120 or lover 
but beyond that point little relationship was present. Draw 
a scatterplot that night illustrate such a finding. 



Creativity . . 

♦ I f I '■ I ■ - 1 ,■ 

80 90 ICO 120 130 140 

Intelligence (I*Q.) 



Biserial Correlation . Occasionally an Investigator wishes 
to determine the relationship betr<^en two characteristics, one 
of which is Class A and the other a dichotomous Class C charac~ 
terlstic. The appropriate descriptor of relationship is the 
blserlal r. An example might be the relationship between 
intelligence (Class A) of unemployed adults and their political 
party preference (Democratic or Republican) (Class C). This 
technique may be expanded along the Class C vr^iable to include 
3, 4, 5, or more categories. 

Spearman Rank-Order Correlation . When data variables 
yield Class B data, the Spearman rho is the equivalent measure 
to r. It requires that data varlabl-'s be ordered and is able 
tc accept ordered ties. Excessive numbers of ties tend to 
attenuate rho and should therefore be adjusted. An alternative 
method is the Kendall Tau which is used less frequently but is 
acceptable and to some researchers is desirable. 

Cont ingency Coef f Icient « When both varla'^les are dichoto- 
mous Class C variables, an extension of the Chi-square techniques 
yields a measure of relationship C. This, for example, could be 
applied to our previous Chi-square example of the difference in 
voting behavior of college graduates and nongraduates. In that 
analysis we found we could not reject a hypothesis of independence 
of voting behavior and gradi 2 ate. We could further compute a C 
to Indicate the degree of relationship. We find C « .14. 
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l^tlplft VTiable RftlatAo nahlp» « K«ny studies wish to 
dstsxBlns tho mlstlooship bstussn oas vsrieble sud ssvsral 
other variables. The general form for such studies Is the 
Multiple Regression Analysis model. It requires measures of 
all aubjeots en m 11 ^esmsds that the single 

depe&dwit variable and most independent variables be of the 
Class A level. The regression model fits several types of 
studies. It yields a multiple coefficient of eorxelation» &; 
a prediction equation, partial coefficients of correlation 
(holding one or more variable constant), and finally permits 
analysis of relative contribution of each independent variable 
to the prediction of the dependent variable. The Investigator 
is cautioned against using large numbers of independent variables 
in a study with this model. When the nuiBber of independent 
variables (predictors) equals or exceeds the number of subjects, 
the solution is overdetermined and R always is equal to unity. 

An accepted rule of thtmb is to limit the number of predictions 
so that R < R/2, where N is the number of subjects. 

Dlscriain^t An extension of the multiple 

regression model yields a model for multiple prediction of a 
dichotomous Class C variable. The sane cautions exist for 
discriminant analysis as do for multiple regrassioA. The 
same types of .studies may be completed - the ooJ' ference 
is that the dependent variable for discrlminas^' /sis is 
dichotomous rather than continuous. 
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We need to know how well the ideas and issues in this mnual are 
communicated to you. You are the test audience for this material. 
To remove or strengthen the weak spots, to retain or improve the 
stron® ones (if any) , an account of your learning experience ^ 
you read these seciions is crucial. Use this sheet as you stuay 
S5d T^Tdown your reactions. Don't be concerned with typographical 
errors. We're concerned with how the message is coming through. 

SECTION # TrrLE__ — 



Give topic, 
paragraph or 
sentence, page 



Importance 

Rate 

1 2 3 4 5 
Low - High 



Clarity 

Rate 

1 2 3 4 5 
Low - High 



Suggested 

Improvements 



Over-all Rating of the Section: 

Important 

1 2 3 4 5 

Very Very 

Low High 

(Circle one) 



Uce a 5-polnt Scale 

Understandable 

12 3 4 

Very 

Low 

(Circle one) 



5 

Very 

High 
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Proposal ITritiag 
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The acid test of Aether you have attained the objectives of 
this Institute Is the writing of an adequate proposal* A fair 
degree of coopetence In the previous eight sections is a prerequi- 
site* If you can't specify behavioral objectives* or can't select 
an appropriate design and tools for analysis* then resign yourself 
that any proposal you conceive will be an abortion* 

But* a healthy proposal needs more than mastery of the previous 
material* The "more" consists primarily of certain communicative 
skills* The architect may have created a great design; the blue- 
print specif lcatior4S may be accurate* clear, and detailed; the 
contractors capable and realistic In their estimates; etc** but 
somebody has to nay for the building costs* The analogous task 
is to convince the bank that this is a needed and worthwhile 
investment* 

Few researchers can* out of pocket, fund the kind of research 
they want to do* The decision to fund or not will be made by 
others; and, in most Instances* the decision will be based solely 
on the proposal* However, viewing the proposal as a mercenary 
medium should not obscure its other functions* The proposal can 
also serve: 

1* to clarify ideas and detect limitations in 
your own thinking, 

2* to communicate to peers and thus instigate 
useful comments and jolly criticisms, 

3* as a first-draft report of the completed 
study* A sound proposal requires little 
more than a change of tense, a fllling-ln 
of data actually collected, and anpllfied 
discussion to become a complete report* 

Most of us think proposals ere good training devices for graduate 
students but not for ourselves* They are hard work* 

A proposal* being a detailed plan for a research study, is 
addressed to three general questions: 
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1. I'Jhy does this study need to be done? 

2. Hov 7 are you going to conduct the study? 

3. What win be the effects of the study \dien 
completed? 

Proposal formats represent an analytic outline of the issues in- 
volved in answering these three functions* Required formats vary 
from agency to agency; some demanding an exhausting degree of 
specificity, others relatively informal. In the following dis- 
cussion, the outline contains those components generally required 
in any proposal* 

I have assumed that you have read and understood (to a 
moderate degree) the previous eight sections of this manual. If 
you are not familiar t^lth the usual terminology in measurement, 
design, and statistics, you may need to refer back to the appro- 
priate section. As a self check, you should be able to give at 
least a verbal definition for the following concepts: 

randomization null hypothesis 

control operational definition 

stratified sample an experimental study 

Failure Indicates a need for help. (Help for this instruc- 
tional system.) 

Objectives of This Unit 

1. To Identify the major compouentc. of a reserxrch proposal 
their function and criteria 

Desired behavior: Recall an ordered set of 

proposal components with 
criteria for each. 

2. To detect and remedy certain typical weaknesses in 
proposals . 



Identify weaknesses In 
examples and indicate 
change, required to approach 
criteria 



Desired behavior: 



3« To coQsCmieti a sound proposal outline In & problen area 
of your Interest. 

bsbavlor* Wrlta a 30 od proposal* Kst 
It funded, and let ue knou. 

The Target Audience 

Most proposals are sent by the funding agency to a board of 
reviewers. Each of the reviewers Independently analyzes the 
proposal and makes a reconanendatlon. Usually 9 be reports reasons 
for his decision, and ^thlu the recommended for funding cate- 
gory may rate the proposal for priority or desirability* Decisions 
of reviewers are followed by the funding agency* 

Since the reviewer plays such a crucial role, what ore some 
relevant characteristics of the typical reviewer? 

1, He is not a specialist in your particular 
problem area. He will depend upon you to 
ac<]ualnt him in a succinct fashion with the 
importance of the study and the issues Involved* 

2. He is extremely busy* He needs to find the 
meat of your proposal quickly. Obscure and 
ponderous writing are high on his blacklist. 

3* He knows research methodology. Weaknesses 
and omissions will be readily detected, e.g., 
you can’t fool Idn by skirting or Ignoring 
a control problem. 

4 . He has an extensive knowledge of other proposals 
(good and bad) and of conflated research. There 
is competition for most researdi funds > and the 
quality of the competition Is increasing. Your 
proposal will have to rank high among those In 
his in-basket. 

Examine the outline in Figure 1. It illustrates the sequence 
of topicSf and over-all function of each, needed for a typical 
research proposal. As you read, think of research ideas of your 
own and relate them to the components of this outline. 



The top components of the proposal will be discussed in the 
order of the outline. Each discussion will amplify the function 
of that component for the reviewer^ will present some objectives 




Topic 



Function in Terms of General 
Questions Answered 



1. Introduction 

Statement of the problem 

Review of literature 

Objectives 

2* Procedures 
Design 

Sampling 

Measurement 

Analysis 

3. Product and Use 

4c Personnel and Facilities 
5. Budget 



What needs to be done and why? 

What has been done that Is 
relevant? 

What are the specific test- 
able goals? 



What Is the structural plan? 
What control will it afford? 

What population will be 
sampled? What size saamle 
and how drawn? 

What will be measured? How? 

How will data from the meas- 
ures be eKamlned? 

What will be the end-product 
of the study? 

What contribution could It 
make? How? 

Who will do the study? What 
is their relevant compe- 
tence? 

Wh.it will each part of the 
study cost? 



Figure 1. The Major Components of the Research Proposal 



or criteria that the component should attain; and Hat some 
couBnon weaknesses* An Itemized count of proposal weaknesses* 
based upon a study of many proposals* is contalr^d In Appendix C. 

The Introduction 

I. Statement of the problem , A terse* clear statement of 
the problem is a good opener. The object of this section Is to 
present the nature and inport^ance of the problem to the reviewer. 
The section consists of two parts: the major part* a description 

of a need; and tbt minor part* a brief preview of how the proposed 
study will relate to the need. The need for the study may rest 
upon its probable contribution to knowledge or Its action upon 
society sometimes both. If this section falls to convince the 
reviewer that the study Is worth doing* then the entire propdsal 
falls. 



a. Criteria for judging the significance of the problem 
include: 

(1) Ceserallzabillty - either In terms of theory* 

or to other populations* or to a range of practical 
problems* or a demonstration to be emulated* etc. 

The fantasy of funding agencies lightly tume 
toward eternal monuments to world-wide educational 
progress flowing from each small contract. Soberly* 
they want to see that y^^i affect something more 
than your own comer. 

(2) Theoretical contribution 

(a) testing* expanding* or qualifying previous 
theories. 

(b) adding to, creating* or otherwise contrlbutlug 
to new theory. 

(3) Eaptrlcal contribution (similar to (2) above* except 
relating to observation) 

(4) Practical contribution - relating to a critical 
social need and its solution. 

(5) Methodological contribution - creating* refining, or 
extending an instrument or technique. 
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(6) Innovativeness this criterion is part of (2)» 

(3)» (4)» and (3) but is periodically eisphsslzed 
by many agencies. 

(7) Panic points - "do it now or it will be too late." 
Either the opportunity to gather the data will be 
gone, or the problem will have grown unmanageable, 
or costs will skyrocket, etc. 

b. The common weaknesses in the research problem are: 

(1) The probUji does not strike the reviewer as signifi- 
cant. Probably, the description of the need has 
failed to grab him vitally. Some problem areas are 
of great significance, but the aspects stressed in 
the proposal are among the trivial variables Involved. 
How can you determine the most relevant varl...le8 of 

a problem? One technique is to list them by such 
classes as: learner characteristics, teacher char- 

acteristics, instructional methods, etc. A compari- 
son may shed light. Previous knowledge of both 
theory and practice are yardsticks, and the criteria 
mentioned above. 

Clarity of expression is crucial here. Try 
out your presentation on colleagues. And uot just 
old pals. 

(2) A second error is presenting the problem is 
inflation. A small grant contract will not erase 
the woes of education nor will all the funds in the 
agency. Reviewers know this well. You can demon- 
strate your awareness by limiting the problem to 
treatable or testable scope. If this appears to 
involve a conflict between generality of the study 
and manageable limits, cut it down, then think about 
implications • 

T. i slant of the proposed study may be Indicated 
in a few sentences. Inclusion of an overview of 
the research direction and definitions of any esoteric 
terms or abbreviations to be used subsequently will 
warm the heart of the reviewer. 



o 
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The following two paragraphs are from the problem section of 
a research proposal. 

Personality evaluation has been a problem of concern 
to psychologists for many years. The science of psychology 
has grown through our knowledge of the ways in which man 
reacts to his environment. This growth, along with man's 
quest to explain natural phenomena has led some men to 
make atteoqpts to find some way to describe human reactions 
and to make predictions, even if tentative in scope, of 
how certain persons will react when faced with specific 
situations. 

Knowledge of human behavior has been gained through 
the many Investigations made throughout the history of 
the science. This knowledge has taken numerous forms, 
primarily as a result of the divergent "frames of reference" 
used by the investigators in their attenqpts tc explain 
human behavior. 

List any criteria met or weaknesses in the above statement of a 
problem. 



Related literature . This section should establish the 
base from which your study moves. Summarize the pertinent research 
or practices, evaluate them, and demonstrate how your project 
relates to them • particularly, how it contributes to progress in 
its specific field. This is the logical first step in proposal 
preparation. Often, a thorough search will uncover a new perspec-* 
tive to the problem or a refinement of your original approach. 
Typically, the search means digging through professional journals 
in the library. Some hints about library and other sources are 
given in Appendix B. 
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Functions of this section Include: 



a. r«.ovlde the non-speclallst reviewer with a succinct 
overview of the particular area. 

b. Offer some evidence of your scholarly skills. 

c. Reveal that you are aware of recent developments. 

d. Delineate how your study springs beyond previous work. 

A lengthy list of titles or names Is of no use to the reviewer. 
He has encountered fluent name droppers before. Instead, focus 
upon a few select references, really relevant to your study, accc:^- 
panled by a critical analysis of their methodological limitations. 
This will Illustrate some of your research skills as well as your 
knowledge of the area. 

Two kinds of assertions to be avoided as the black death are: 

"No relevant work has been done In the area." 

or, "So much has been done and written In this area that 
a summary Is impossible." 

Reviewers feel that the authors of such statements should never 
have been allowed out of graduate school - or never admitted. 

List functions fulfilled and/or weaknesses In the following 
excerpts from Related Literature coiiq[>onents : 

(1) When compared with other areas of learning the 

study of concept formation is retarded. The psycholog- 
ical literature reveals a paucity of empirical studies 
of thinking. Yet thii.klng is a topic in which educa- 
tional psychologists have a strong Interest. 



(2) The early work of Thorndike and Edling (1912) explored 

this area. Hull (1941) presented his stltrulating theory of 
the phenomena and followed this by his later extension to 
motivation. Relevant research has been done by Dllllnger 
(1950), Twelker (1949), and Welch (1952). Of course, all 
students are familiar with the review of the field by 
Hlckock (1938)... 



3. Objectives . If the proposal is primarily directed toward 
an extension of knowledge, this section will he focused upon 
questions or h 3 rpotheses. A more applied or practical direction 
may shift the focus to products. The reviewer will be concerned 
that the objectives are achievable. 

One essential requirement is that they be testable . If no 
possible outcome of the study can refute the objective or hypothe- 
sis, it Is not testable. The research procedures must bear on the 
truth or falsity of stated hypotheses. This Is the main reason 
behind our reiterated emphasis upon operational definitions and 
behavioral objectives. They constrain concepts and goals to 
face the evidence. 

Hany important research topics involve covert processes or 
categories that are not directly observable, e.g., love, Intelli- 
gence, thinking. It Is certainly legitimate to construct parallel 
lists of objectives: a theoretical and an observable set, the 

latter derived from, or the Implied effects of, the former. In 
a research proposal the observables are a must. The previous 
sections on Behavioral Objectives and Measurement present a more 
adequate treatment of these issues. 

In areas where theory or prior observations are minimal, 
questions are an appropriate form. As knowledge Increases, 
hypotheses become more appropriate. Questions remain the 
typical objectives of survey studies: How many? What Is the 

relationship between? However, the more specific the question 
the more It reflects the preparation of the researcher. 
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Dleregard the nonsense frequently written about stating 
hypotheses in a "null” or no-difference form. If you have 
hypotheses, state them the way you expect the results to go. The 
null Is an alternative (or set of alternatives) which is logically 
xnaispensibie in analyzing your data. It belongs in the results ’ 
section, not here. 

The objectives should flow from your statement of the problem. 
You have stated a needo Now, in what wa^ are you contributing to 
filling that need? The reviewer will then look to see if you have 
set up procedures to attain the objectives. In the methods or 
procedures section there must be a step-by-step account of \^at 
will be done to approach each objective. 

Common flaws in this section includes 

a. Pie-in-the-sky objectives. Vague, global goals 
ii!q;>ervious to the study's outcomes. 

b. Sandbag objectives, not referred to or dealt 
with by the proposed procedures. 

c. Hidden objectives, contained somewhere in the 
text of the proposal. An astute and dilfgent 
reviewer can find them he takes the tiL<e. 

List requirements fulfilled and/or weaknesses in the follow- 
ing objectives: 

(1) The purpose of the investigation is to show that the 

unobserved and unmeasured behavior of high school students 
is different from their behavior when it is observed 
measured . 



i survey is to find out what proportion 

of of Neeknak High School are earning less 

than $1,000.00 annually. 
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The broad hypothesis Is that a procedure cam be followed 
which will leac to the Initial formal at ion, re^'isioI!l, and 
final development of a broadly conceived theory of education 
based up on sociological and ether relevant research findings* 



Procedures 

This section presents a detailed explanation of what you 
are going to It, above all other sections. Is cast in behavioral 
terms and operational definitions: a concrete description of 

the project. If the reviewer gets this far, he will devote more 
attention to your procedures than any other section for two reasons: 
one, he can easily discover from this down-to-earth section just 
what the study may accomplish; two, procedures are a spawning 
bed for numerous species of error. 

If possible, a short introductory paragraph indicating the 
over-all kind of design and the sequence of steps you will take 
is helpful. 

1. Design . If you are using a design with a conventional 
name, so label it and then explain how this situational plan handles 
the independent variables and provides for control. The term 
"design" in research literature refers to the structure of an 
e' perlment rather than to a non-experlmental study. You are probably 
almost painfully aware that the previous materials are strongly 
slanted toward experimental studies. Evidence from an. experiment 
is more highly regarded than evidence derived from other approaches. 

However, not all funding sources are looking for, or require, 
experimental approaches. A discussion of the relative merits 
of non-experimental techniques may be found in Kerlinger (1964) . 

If your study is not experimental, i.e. correlational, case study, 
survey, etc.. Indicate its category and give especial emphasis to 
control measures that you will use. 

Check back to your statement of objectives. Lay out the 
steps to answer each question or to test each hypothesis. A 
summary chart with questions jr hypotheses in one column and the 
procedures relevant to each in an adjoining column may help. 
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The design description must account for any variable that 
might influence the results of the study. Indicate how you will 
control it, e.g, build it into the design and measure its effect , 
block it off, incorporate its measures into your toois of 
analysis, randomize, etc. A useful discussion of the kinds and 
relative merits of control measures may be found la McGulgan (1960) . 

No design affords perfect control. Some factors will remain 
as possible sources of contamination. It is the degree to which 
these are strained out that marks the better design. Reviewers 
tend to be more favorably impressed If you demonstrate your aware- 
ness of the weaknesses cf your study. At least itemize the 
loopholes and indicate why you were unable to control for them. 
Sometimes you can point to cost, i.e. the enormous number of 
requisite control groups, time and effort required, etc. The 
reviewer usually has broad experience with such limitations. 

But, he will prefer that you point out the weak spots and why 
they remain* 

As a reviewer, comment on this proposal design: 

A osychologist proposes to test the hypothesis that 
early toilet training (Head Start) leads to a t3^e of 
personality noted by compulsive cleanliness; conversely, 
late toilet training leads to sloppiness. Previous 
studies have shown that middle— class children receive 
toilet training earlier than do lower class children. 
Accordingly, he asserts he will select two groups, one 
of middle-class and one of lower-class children. He 
will give both groups a finger-painting task and compare 
the amount of smearing and how many times they wash the 
paint off. 



If you have cited any design weaknesses, point out how to correct 
them. 



2 • S angllng and measurement. Because you vis5i to genera- 
lize any findings beyond the particular subjects in 
and/or beyond those particular days and places when the investi- 
Kfttion occurred, most research studies are perceived as 
operating with a sat-ple and generalizing their conclusions to a 



This target p£?pulation must be adequately described. The 
reviewer should be able to tell readily who is included and who 
Is not included in the population. To a large extent your choice 
of sample determines the population to which the study applies. 
Logically* the sequence would be to define the population and 
then specify how a random sample* or some modification of a random 
sample* would be selected. In Instructional research you are 
often constrained to study the students available, so that you 
must construct the relevant features of the population from the 
characteristics of your sample. List those subject character- 
istics i^ich previous knowledge indicates may influence the 
behavior you are measuring. Omit characteristics whf.ch make no 
effect on the measure. Color of eyes (within the normal range of 
colors) rarely affects measures of learning. ?: 



In theory* in the proposal* and actually, you will probably 
define the population first - in terms of relevant characteristics 
- that is, characteristics probably having measurable effects on 
the dependent variables. Then, show how you will select a sample. 
The reviewer will be concerned whetLor yo»^r method of sampling 
tends to systematically bias the measures you take. Point out 
your efforts at randomization or stratification. If you have to 
use intact classes* schools* or other inclusive groups be sure 
you have clearly stated this. 

Random sampling procedures are highly rated by reviewers and 
for most purposes yield logically defensible results at relatively 
low cost. Meticulous reviewers tend to look for 3 kinds of 
randomization procedure (in order of priority) : 



(1) Selecting the subjects from the population 

(2) Assigning the subjects to groups, e.g. In a form 
group study placing them in Group I, XI* HI ^ 

(3) The, and this Is a separate step, assigning the 
the treatments called for in your design to the 
groups . 

Generally, your sampling concerns relate to the peopl e whose 
behavior you are studying. Qccasionally you may have an interest 
in sampling from a variety of treatments. Get some design consul 
tant assistance on this. It does offer power in the conclusions 

attained. 



. Si^IE SIZE* Specify the size of eac! group in the study* 
Three factors should be considered in determining the number 
of subjects Involved* 



1 . Tllfi rffhfMPoCf f\f T4P 

you desire or demand that your instruction or 
treatment makes a noticeable effect on a small 
group (s:iy 20 students), use small groups* 

If it makes only a tiny effect, you can meet 
all statistical criteria but you will need 
larger groups* 

2* The cost involved* Larger numbers, if not 
easily available, may exponentially inflate 
your budget. 



3* Statistical considerations. Previous know- 
ledge of any consistent effect can be trana* 
late^ into the sample size required to reach 
the desired statistical conclusion - but 
only when the effect Is known to be In the 
appropriate direction and its extent as well 
as an estimate of variability has been 
i measured* This Is one by-product of pilot 

i studies* Almost any statistics book or 

graduate student in a statistics course can 
show you the translation procedure. 



If possible, plan to use equal numbers in each group* If 
you |:an*t assume equality, get some consultant help (statistical) 
to pjliit out how you will deal with unequal nusd>ers* Reviewers 
like] to see that you have worried about this* 



• SCHEDULE* A realistic time schedule adds weight to your 
proposal* Graphic or flow chert representations are easiest for 
the pviewer to follow* Large scale propose'ls are often acconq>anled 
by a’PERT chart* This system. Program Evaluation Review Technique, 
takeh a bit of study but has a high payoff in increased accuracy 
of your planned sequence of actions* 



If you are not experience J in the kind of research you pro- 
pose, allow more tine than you anticipate* Copies of materials 
and instruments will refuse to appear on time, subjects will hide 
or get busy with life or death urgencies, your assistant will 
misread the instructions and you'll he searching for a new group 
of subjects as well, etc*, etc* And remember, all funding; agencies 
require a report* This takes secretarial and duplication time* 




Usually the funding agency will indicate, in advance, a 
starting date if the project is approved* Allow for a delay 
even In this* 

3* Analysis . This component describes how you will examine 
the data to obtain evidence bearing on your objectives* The 
appropriate examination tools are determined in part by the 
objectives, in part by the design, and in part by the class of 
data* The previous sections of this manual, VII and VIII, are 
useful guides. (1) Name the analysis tool, (2) show why it is 
appropriate, and (3) indicate how the product of the analysis 
will bear on your objectives* 

If you have a proposal constructed to this point, and are 
still a bit cloudy about the appropriate analysiSs seek help. 

In a small institution you can usually find some staff member 
in mathematics, psychology, economics, or even education who 
can and will help for the price of coffee and a sympathetic 
ear to his troubles with the administration* If possible, seek 
out one who has been teaching statistics regularly and recently * 
If you can't find local assistance, call the nearest Regional 
Office of the USOE. Addresses are in Section X. They will 
refer you to the nearest specialist or provide direct assistance. 

This is the place to set out the null hypotheses. A useful 
technique is to place the research hypothesis of interest in 
one column, the alternative null in another, and In a third 
column, the kind of result from the analysis that will indicate 
rejection of the null to be the logical decision* 

Frequently you will be unable to cite the appropriate tools 
because unknown, as yet, dimensions of the forthcoming data will 
determine what tool to use* Spell out the contingencies as you 
see them* Show alternative plans of analysis* Reviewers wax 
warm and beneficent when they perceive you have anticipated 
problems and have planned coping strategy* 

Product and Use 

The purpose of this section is to point up what contributions 
the completed study will make and how. The final report itself 
may be suitable for dissemination through other channels. A 
journal article may be a reasonable expectation. 

Both effects on the body of knowledge and on working practi- 
tioners may be estimated* Suggest how dissemination to the 
latter may be effected. Often the study will generate by- 
products such as evaluation Instruments, instructional materials, 
films, etc. These may be highly useful to others. 
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Perhaps a significant contribution to your am institution 
is a likely yield* Even the effects on the research team or 
the individual researcher in terms of professional experience 
and increased capabilities. 

But, keep it brief. 

Personnel and Facilities 

Name the people who are to work on the study. Briefly 
indicate experience relevant to the research. If you 
haven't any, cite attendance at this Institute. Evidence 
of interest in pursuing this kind of research is appropriate 
if experience is lacking* Don't deliver a eulogy to a 
bunch of nice guys. Re^ewers loathe nice guys. 

As a last ditch resource, consider acquiring the 
services of an experienced researcher as a consultant to the 
project. He will insist on being paid and this will add to 
the budget. Spell out clearly Just how he ^11 serve the 
project.. Retain the decision function and allow him to advise 
and recommend. 

If you have special facilltir? that will aid or are required 
by the project. Hot them. Libra ;^.es suitable for graduate work, 
computers, training programs, secretarial services, and office 
space may be relevant. 

Budget 

The easiest way to get a feel for proper budgets is to 
peruse some approved budgets for projects granted by the same 
agency* The agency will have sent you as part of the appli- 
cation form, probably a rather detailed budget schedule* Use 
some worksheet facsimile and construct a sample budget* Show 
it to an experienced researcher. Usually, he can quickly 
scan it and give you some useful suggestions* Your own fiscal 
officer may help* The institution may have to contribute 
costs you hadn't anticipated unless you provide for this* 

Show each facet of your institution's contribution from staff 
time, secretarial services, materials, etc. 

The usual error of first efforts at budget construction is to 
underestimate and omit* If your procedure is detailed, the 
description can be used as a guide. Check every step including 
preparation of the final report. 



Don't be overconcemed with this component. Budgets on 
approved projects are often changed through negotiation with the 
funding agency. Consider the proposal budget as a good first 
draft o Many first rate projects have to be re-budgeted later. 

Selecting the Funding Agency 

Supporting sources are so numerous that this entire manual 
could not list them. A guide to sources of Information about 
funding agencies is contained in Appendices C and D of this 
section. Section X contains some guides from the United States 
Office of Education your most probable funding source. 

For your first project, don't overlook your own institution > 
The president of a small college can often dig up a little 
seed money for a pilot project. 

Agencies will send you their requirements, formats, and 
usually some Indication of their interests. Often a short 
letter outlining the project in an informal way will help 
them to decide if it is worthwhile to submit a detailed proposal. 

The Research Marketplace 

The preference of agencies for different kinds of projects 
changes over time. I have stressed the orientation that if you 
are interested in a problem area, work up your ideas; then locate 
the appropriate funding source. A contrasting approach is 
followed by some researchers. These social science street 
walkers try to sound out agencies as to preferences, then 
write up proposals in that area - wherever the demand lies. 

Some compromise with reality demands is Inevitable. However, 
the increasing multitude of funding sources lends assurance to 
the idea that a good proposal will get supported (sooner or 
later) . 

What To Do When Your Proposal is Rejected 

After you have decided to change your specialization, 
written and tom up several suitable rejoinders, etc., find out 
why they turned it down* Most agencies will sent you a summary 
of reviewer comments. Evaluate them* If the critiques are apt, 
re-do the proposal. Often the agency will indicate if they 
think re-submission is suitable. Or if the proposal looks 
good to you, submit it t'' inother agency. Some good projects 
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have gone through three rejections from different agencies » 
then been f unded» and subsequently acclaimed as hallmarks of 
progress. To gain perspective, talk to any publishing author o 

Good luoko A coBiplete proposal Is contained In Appendix 
A. This was approved, receiving high ratings frcm reviewers, 
despite some weaknesses (which will be obvious to you) • 
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APPLICATION TO THE CCjf4MISSI0NER OF EDXATION, U. S. OFFICE. OF EDXATION 
DEPARTMENT OF HEALTH, EDXATION, AND WELFARE, FOR A GRANT TO SUPPORT 
A RESEARCH PROJECT UNDER THE PROVISIONS OF TITLE VII OF THE 
NATIONAL DEFENSE EDXATION ACT OF 1958 (P. L. 85-864) 



Submitted TEACHING RESEARuri 

A Center for Research on Teaching 
Oregon State System of Higher Education 

Address I Monmouth, Oregon 



Telephone number » SKyline 7-1421, Ext, 73 

Initiated bv t Dr. Bert Y. Kersh, Associate Research Professor 

TEACHING RESEARCH 



Transmitt ed bv t Dr# Roy E# Lieuallen, President 

"" Oregon College of Education 



Date : July 26, 1961 



Bert Y. Kersh 



Roy E. Lieuallen 



I, Project Title * 

Directed Discovery vs. Programed Instruction; A Test of a Theoretical 

Position Involving Educational Technology* 

n. Problem * 

Increasingly more importance is being placed on the process of learning by 
’•directed discovery’’* Representative of this increasing emphasis is the newer 
techniques and materials presently under development by the UICSM (University 
of Illinois Comnittee on School Mathematics) directed by Professor Max Beber- 
man* By requiring the learners to discover for themselves new relationships 
in mathematics, it has been demonstrated that high school students are capable 
of learning advanced concepts more effectively than otherwise* The act of 
discovery itself is considered to be the primary reason for the success of the 
new mathematics curriculum (Beberman, 1955; Hendricks, 1961). 

Paradoxically, increasingly more Importance is also being placed on the process 
of learning by programed instruction ("teaching machines")* "Discovery" by the 
student is minimized in programed instruction which characteristically presents 
the material in such small steps that the learner does not have to search far 
to find an answer. This is especially true when prompts and cues are added. 

Yet, the research evidence to date indicates that programed instruction is at 
least as effective as more conventional methods of instruction and may be far 
more efficient in terms of teaching time. 

The fact that both the "discovery" and the "programed" instructional methods are 
presently in the forefront of attention serves to revive an older, unresolved 
research problem in a new context, and with greater potential for solution* The 
problem is to determine which process of learning is superior, a highly directed 
(formal) learning which places the learner in a position of complete dependence 
on the teacher, or non direoced (informal) learning in which the learner must 
rely almost completely on his own cognitive capabilities* Advocates of the non* 
directed, informal (hereafter called "discovery") process claim a n^ber of ad- 
vantages, most of which are included in a recent article by Bruner (1961)* Bruner 
hypothesized that learning by discovery benefits the learner in four ways; (1) it 
increases the learner’s ability to learn related material, (2) fosters an inter- 
est in the activity itself rather than in the rewards which may follow from the 
learning, (3) develops ability to approach problems in a way that will more likely 
lead to a solution, and (4) tends to make the material that is learned more read- 
ily accessible in memory - tliat is, easier to retrieve or reconstruct* 

Research evidence is not entirely supporting of the claims of Beberman and 
Bruner* The more recent evidence suggests that learning by discovery does not 
necessarily benefit the learner directly in terms of retention and transfer, but 
it does foster interest in the task (Kersh, 1958). Vilhen interest is generated, 
the learner tends to continue the learning process autonomously beyond the formal 
learning period* As the results of his added experience, the learner then raises 
his level of achievement, remembers what he learns longer and transfers it more 
effectively* However, sufficient interest to continue learning evidently does 
not appear unless the learner expends intensive effort without help for an ex- 
tended period (Kersh, 1961) 

In other words, the evidence referred to above suggests that learning by dis* 
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covery is superior to highly directed^ formalized learning only in. terms of in* 
creasing student motivation to pursue the learning task* Learning with direc- 
tion is far more efficient in that the $tudsnt learns more during a given per- 
iod of time and, when the direction is pertinent, comes to understand what he 
learns more completely. Obviously, programed instruction may be allied with 
highly directed, formalized teaching li^athods. 

The explanation for the elusive drive generated by independent discovery is not 
evident, but several have been offered, including the Zeigarnik effect of sup- 
erior memory for unfinished tasks and the Ovsiankina effect of resumption of in- 
complete tasks (Kersh, 1961). Neither of the latter is entirely adequate be- 
cause they simply describe the conditions under which the motivation may be ex- 
pected to appear. Actually, the motivating effect may appear even when the tasks 
are completed. Another explanation, and the one which is of primary concern in 
this present proposal, is that tHe' motivating effect is learned through a process 
of operant conditioning. By this theory, the learner, yAxo is forced to discover 
the solutions to problems without ^ielp, engages in a kind of behavior often de- 
scribed as "searching**. Ihe searching behavior is reinforced by the teacher who 
is monitoring the learning process, and by the learner *s own successful progress, 
towards a solution. This explanatii^ would support the claims that Bruner makes 
regarding the increase in the learner's sibility to learn in related areas, to 
solve similar problems, and to continue the searching behavior beyond the formal 
learning period whether or not there are any extrinsic rewards ‘involved. The 
theory. also fits well with the research evidence i^ich suggests that the .aotlva- 
ting power of learning by discovery does not appear unless the learner engages 
in such searching behavior over a relatively long period of time. 

If the above explanation is true, then it should not be necessary to rely com- 
pletely on the less efficient, time-consuming discovery techniques* In short, it 
should be possible to employ the techniques of programed learning to achieve the 
same desirable benefits which may accrue from learning by discovery. Many of the 
techniques of programed learning are based on the operant conditioning model 
and, through the use of appropriate educational media, are adaptable to any tea- 
ching objective which may be translated into behavioral terms* 

The purpose of this present project is to conduct an experiment to test the 
theory that a teaching (machine) program which is specifically designed to de- 
velop and foster the kind of searching behavior that characterizes the discovery 
process, will produce equally as well the desirable benefits to the learner which 
are usually attributed to discovery techniques alone* 

If the experimental evidence substantiates the theory, it will warrant further 
study* The development and eventual application in the classroom of efficient 
and effective techniques for using the new media associated with programed learn- 
ing will no doubt follow* 

III. Oblectives 

The present project has two objectives: 

1. To develop a "programed" modification of an existing unit of instruc- 
tion (The Associative Law) modeled after the UICSM "discovery" method, and 
adapt it to a group-pacing technique which provides individualized feed- 
back to the learner (see detailed description under Procedure* below). 
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2. To conduct an experiment with the programed method designed to test 
the following hypotheses t 

a* Students taught by the programed modification (Objective 
achieve the learning objective more rapidly than students taught b 
the discovery method. 



students who are taught by the discovery m<?thod. 

c. Within a period of eight weeks following the formal learning per- 
iod, students taught by the programed method are able to recall the 
learned material from memory and apply it in the solution of mathe- 
matical problems as well as students who are taught by the discovery 



d. In a task of new learning following within 24 hours after the for- 
mal learning period, students taught by the i»ogramed method reveal by 
their written work and self reports that they employ the same or the 
equivalent techniques of independent discovery and problem solving as 
students taught by the discovery method. 



The project will be conducted in two phases: (l) materials development phase, 

and (2) the experiment. During the materials development phase, the necessary 
modifications to the discovery materials will be produced and the techniques 
developed for using them. During the same phase, the teachers for the exper^ 
mental classes will be trained In the use of the materials and techniques. The 
first phase is expected to last approximately eight*months. A more detailed de 
scrlption of the programed materials and techniques is given below. 



Deslan. Two experimental groups of elementary school children (grade 5) 
will be taught the Associative Law in different ways: one group by the Dis- 
covery method, and the other by Programed instruction. Two different in- 
structors will be involved. Then the experiment will be replicated with 
two new groups of children. This time the same two instructors will teach, 
but they will exchange methods of instruction. 

The counter-balanced design is diagramed below: 




method 



IVi Procedures 



The Experiment 



Instructional Method 



Discovery 



Programed 



Teacher A 



Teacher B 




Teacher B 



n = 15 



n = 15 



Teacher A 



n = 15 



n = 15 
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Selection of subjects . A total of 60 fifth graders will be selected from 
the MoraKOuih Elementary School (and neighboring schools) on the basis of a 
pre-test of their knowledge of those number concepts and arithmetical op- 
erations that are pre-reguisite to learning the Associative Law* to bet- 
ter insure that the children will be capable cf learning the reguired con- 
cepts, their records of scholastic aptitude and achievement in arithmetic 
will be examined. Only children within prescribed limits of ability and 
achievement will be accepted. Thereafter, the group will be sub-divided 
into four groups of 15 by random processes, and each group will be assigned 
to one of the two experimental groups. 

The actual instruction will take place on the campus of Oregon College of 
Education. The physical facilities will be the same for each group with 
the exception of the special equipment used with the Programed Group. 

Experimental techniques . The Programed Group will be taught in the spec- 
ially equipped laboratory classroom of Teaching Research, called the 
Teaching Research Laboratory. The laboratory is equipped with a 25-student- 
station TELETEST Communication System (see appendix). This system will 
permit the classroom instructor to present programed materials to the class 
and to provide individualized feedback to each student ifWediately after 
the student signals that he has made his response. The fact that each 
student has made a response and the particular response made is recorded 
in coded form immediately on an IBM (International Business Machine) type 
card at the instructor’s station. The instructor, therefore, has the cap- 
ability of determining, at any point and time, precisely v*at proportion 
of the class is ready to go on, and which individuals are progressing sat- 
isfactorily and which need help. 

The programed method of instruction, using the apparatus described above, 
will not be the self -instructional, individually-paced approach which typ- 
ically characterizes the teaching machine. This present technique uses 
the programed materials as an integral part of the more conventional ap*- 
proaches to classroom instruction. A teacher is present at all times, and 
group-pacing techniques are used. The teacher conducts the class by pro- 
viding information, assigning problems, interacting with individuals, etc., 
as required. The unique characteristics of the proposed programed method 
are the following; (l) The teacher will follow a ^’program” of questions, 
problems, presentations, etc., which will employ such techniques as "van- 
ishing” cues, small steps, branching, etc., as may be appropriate. (2) Each 
student will record his needs and reactions (e.g., "progressing satisfact- 
orily”, "need help", "repeat”), or his soUJtions to problems through the 
use of the communication system described above. (3) The teacher will 
receive and collate the students* reactions to questions ana problems al- 
most immediately and adjust his program accordingly. (4) The program of 
problems and tasks will require the students to employ problem solving 
and discovery techniques with increasing frequency and on increasingly 
more difficult tasks. In the beginning the program will be highly dir- 
ective in "teaching" the students principles for solving mathematical pro- 
blems and using the inductive method. 

The Discovery Group will be taught by techniques and materials modeled 
after the UICSM course of study. The materials for this present study 
were developed for the fifth grade level at Oregon College of Education 
during 1960 and 1961. 
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The formal learning period is expected to last approximately ten hours 
distributed over two weekSe During the learning period, each subject will 
be brought to the same level of achievement as indicated by pre-estab- 
lished criteria. The learning criterion will be in terms of each student s 
performance on problems requiring the application of the mathematical law 
to be learned. Since both teaching methods use group-pacing techniques, 
all students will complete at approximately the same time. However, in the 
DisGovAry group especially, considerable freedom is allowed the individual 
to learn'at his own rate,* so every student will be given the opportunity 
to demonstrate his achievement on a short written examination whenever he 
and the instructor agree that he may be ready* When an Individual d^on- 
strates that he has achieved the required level, he will be excused from 
further attendance in the experimental classroom. 

The entire experiment will be repeated twice with a three-month time inter- 
val as outlined in the experimental design, above. 

Test of new learning. After each individual student completes the initial 
pf^ram of instruction, and within 24 hours thereafter, he will be taken 
aside individually and given a new task to learn. The task will be that 
of discovering a novel rule for adding a series of odd numbers. The rule 
is usually discovered within a period of 30 minutes. During the new 
learning period, voice recordings will be taken of the subject’s verbal 
report of their thought processes, and any scratch work will be retained. 

Post-test of r .^call . A post-test consisting of problems similar to the^ 
ones used during the learning period will be given to each subject within 
eight weeks following the formal learning period. One third of each group 
will be administered the posti-test two days after the formal learning per- 
iod, another third will be given the post-test after two weeks, and the 
final third after eight weeks. At the same time the post-test is given, 
each subject will be asked to fill out a questionalre on his use of the 
learned mat rial during th Intervening period. They vdll be asked tht> 
number of times the rule was used spontaneously, and the purpose and occa- 
sion in each case. 

Analysis of data . The data will be analyzed by comparing the two experi- 
mental groups in terms of time to learn, techniques employed during the new 
learning task, performance on the post-test, and use of the learned mat- 
erial during the period intervening between the learning period and post- 



Comparisons in rate of learning (Hypothesis a, above //in terms of the 
number of class sessions (or hours) to complex-e the learning task. Chi 
square will be used to test the significance of the obtained differences 
against a theoretical position of equal time to complete. 

Comparisons in average frequency of use of the new learning (Hypothesis b) 
will be based on the questionalre data in which each student will be asked 
to estimate the total number of times he used the information during the 
intervening period (together with corroborating information). If the fre- 
quency data meet the essential distribution and variance requirements, the 
mean differences between groups will be tested by the t test; otherwise, a 
non-parametric statistic such as chi square will be employed. 
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Hypothesis pertaining to memory for the task, vdll be tested with the 
post- test data. The tests will be scored on a pass-fail basis, since the 
concern is with memory for a mathematical law - not with computational 
accuracy or specific procedures which may contribute to overall test var- 
iance. Consequently, the chi square technique may also be employed in this 
analysis. 

The data pertaining to the last hypothesis (d) will be in the form of fre- 
quency distributions of Ihe techniques of problem solving and discovery 
used by the experimental subjects. The differences in the distributions 
for each group will be tested statistically by chi square, if suitable. 

Approximate time schedule . 

The developmental phase of *^he project will last approximately eight months. 

The task of programing involves several try-out and revision cycles, each of 
which Is slov/ and tedious. 

The experiment will require eight months to conduct and two months for analysis 
and reporting. 

Expected end product . 

The experiment will provide new evidence on the relative effectiveness of the 
methods of programed learning and discovery particularly as involves the theo- 
retical position outlined above. 

Publication plans . 

I 

The results will be submitted for publication in the Journal of Educational 
Psychology . A discussion of the methods employed together with their respective 
theoretical basis will be submitted for publication in the Mathematics Teacher 
and related periodicals. 

V. Personnel 

Project Director - Bert Y. Kersh, Ph.D. (Educational Psychology, University 
of California, 1955) present position; Associate Professor on the 
permanent research staff of TEACHING RESEARCH, Oregon State System of 
Higher Education. Research experience: Consultant in Educational 
Research, University of California, 1952-1954; recipient of three re- 
search grants from the Graduate School of the University of Oregon, 
1955-1959; Human Factors Scientist, System Development Corporation, 

Santa Monica, California, 1959-1960. 

Director of Curriculum and Instruction - Vern D. Hiebert, M.S., (University 
of Illinois, 1959) Assistant Professor of Mathematics, Oregon College 
of Education. Related experience: Developed the grade 5 experimental 

materials and techniques at Oregon College of Education which are 
modeled after the UICSM course of study and employ the discovery method. 
Completed a graduate course in mathematics instruction at University of 
Illinois taught by UICSM personnel. Five years teaching experience in 
public schools. 

Experimental teacher I . Vern D. Hiebert, fA.S. (above) 
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Exneriinental teachex U. To be selected from the staff of tte Monmouth 
Elementary School, the laboratory school for Oregon College of 
Education. He will be instructed in the experimental techniques 
by Professor Hiebert, 



Research As sistant - Donald Toevs, M.A., Research Instructor, TEACHING 

RESEARCH, Oregon State System of Higher Education. Related experience: 

_ . . . s ..1 0 st TVi^ - Ve EdliHOa Diroctor 

Kesearcn ass^tsTuant unuer «.t»c w. .. 

of TEACHING RESEARCH, 1960-62. Majored in Mathematics and Education. 



Secretary - Mrs. Neil Amerman, Secretary to the staff of TEACHING RESEARCH, 
Oregon State System of Higher Education. 



VI. Facilities 



The resource personnel, office and ecjuipment of Teaching Research, A Center for 
Research on Teaching, Oregon State System of Higher Education. An agency of the 
State Board of Higher Education, Teaching Research utilizes the full resowces of 
the seven campuses in Oregon’s unified State System, including modern motion pic- 
ture production facilities, advanced electronic computer systems, completely equip- 
ped laboratory schools, and a distinguished faculty of behavioral scientists. 



The resources of the Monmouth Elementary School, the Laboratory School on ^e 
campus of Oregon College of Education. The Teaching Research Laboratory, is lo 
cated in the Monmouth Elementary School. 



VII. Duration 

Total amount of time required: 18 months. 

Beginning: January 1, 1962 
Ending: June 30, 1963 
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BUDGET 




OE 

Category Funds 

Requested 


OSSHE 

Funds 

Contributed 


I. Direct Costs 






Personnel 






Project Director (.25 FTE, 18 tnos. 
1st 6 mo. paid by State, next 
12 mos. Fed. Funds) 


$ 2,700 


1,300 


Director of Curriculum & Instruc- 
tion (.25 FTE, 6 mos.) 


850 




Experimental teacher (one only, 
1.0 FTE, 1 mo.) 


600 




Research Assistant (.50 FTE, 
15 mos.) 


2,400 


1,200 


Secretary (.25 FTE, 12 mos. 


425 


425 


Programing consultant (one only, 
7 days @ $50/day) 


350 




SuDolies and Materials 






Programing materials (mimeo., 
35 mm. film, etc.) 


75 




Instructional materials 




25 


Office supplies 




50 


Other direct costs 






Payroll assessments (6^ salaries) 


419 


175 


Depreciation on TELETEST System 
(Total value of equip. $4,500 - 
Depr. est. for li yrs. for pro* 
portion of time equip, used on 
project) 




100 


Communications 




50 


Travel (for experimental subjects) 


50 




Total direct costs 


7,869 


3,325 


Indirect costs (25^ of direct costs) 


1,967 


none 



$ 9,836 

(cont*d. on next page) 










mm mm 



BUX5ET (cont'd.) 



Summary 

Office of Education funds requested 
Oregon State System contribution 



$9,836 

3,325 
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Total cost of project $ 13,lfl 

Other support 

This proposal has not been submitted to any other agency or organi- 
zation* 

The research proposed herein is not an extension of or addition to 
a project previously supported by the Office of Education* 

Estimated cost to Federal Government ^ fiscal year * 



Category 


Federal Runds Requested 


Fiscal year 1962 


909 


Fiscal year 1963 


8,927 


Total cost (all years) 


$ 9,836 
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i^ppendix B 

Shortcomings found In study-section review of 605 disapproved 
research grant applications, April-May 1959;. All percentages 
are to the base uusuiber 605. 



No. 



Shortcoming 



Class I: Problem (58 per cent) 

1 The problem is of insufficient importance or is unlikely 
to produce any new or useful information. 

2 The proposed research is based on a hypothesis that rests 
on Insufficient evidence, is doubtful, or is unsound. 

3 The problem is more complex than the investigator appears 
to realize. 

4 The problem has only local significance, or is one of 
production or control, or otherwise fails to fall suffi- 
ciently clearly within the general field of health- 
related research. 

5 The problem is scientifically premature and warrants, at 
most, only a pilot study. 

6 The research as proposed is over-involved, with too many 
elements under simultaneous investigation 

7 The description of the nature of the research and of its 
significance leaves the proposal nebulous and diffuse 
and without clear research aim. 

Class II: Approach (73 per cent) 

The proposed tests, or methods, or scientific procedures 
are unsuited to the stated objective 

The description of the approach is too nebulous, diffuse, 
and lacking in clarity to permit adequate evaluation 
The over-all design of the study has not been earexuiiy 
thought cut 

The statistical aspects of the approach have not been 
given sufficient consideration. 

The approach lacks scientific imagination. 

Controls are situter Inadequately conceived or inade- 
quately described* 

The material the investigator proposes to use is unsulted 
to the objectives of the study or is difficult to obtain. 
The number of observations is unsuitable. 

The equipment contemplated is outmoded or otherwise 
unsuitable. 

Class III: Man (55 per cent) 

17 The Investigator does not have adequate experience or 
training, or both, for this research 

18 The Investigator appears to be unfamiliar with recent 
pertinent literature or methods, or both. 

19 The Investigator’s previously published work in this 
field does not Inspire confidence. 



8 

9 

10 

11 

12 

13 

14 



1C 

16 



33.1 



8.9 



8.1 



4.8 



3.1 



3.0 



2*6 



34.7 

28.8 
14.7 

8.1 

7.4 

6.8 

3.8 

2.5 

1.0 



32.6 

13.7 

12.6 
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20 The investigator proposes to rely too heavily on insuffi- 
ciently experienced associates. 5.0 

21 The Investigator is spreading himself too thin; he will 

be more productive if he concentrates on fewer projects. 3.8 

22 The investigator needs more liaison with colleagues in 

this field j or in collateral fields. 1.7 

Class IVj Other (16 per cent) 

23 The requirements for equipment or personnel, or both, axe 

unrealistic . 10 

24 It appears that other responsibilities would prevent 
devotion of sufficient time and attention to this 
research. 

25 The institutional setting is unfavorable. 2.3 

26 Research grants to the investigator, now in force, are 
adequate in scope and amount to cover the proposed 
research. 



Source; Allen, Earnest M. (NIH) "Why Are Research Grant Applications 
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Appendix C 

Where to Obtain Information on 
Fundlna Sources and Procedures 



Guide To Support Pro 5 araiB 8 for Education * 

Education Service Press, Visual Products, 3M Company, 

St, Paul, Minnesota. 1967 (2nd. ed.) Price $12.00. 

This is a worthwhile Investment for the novice. Deals with prin- 
ciple federal laws aiding education, aid from business and foundation sources, 
and provides helpful hints generally on how and where to obtain support for 
research, development, or training programs. 

2* College a nd University Reporter (Topical Law Reports). Commercial 
Clearinghouse, Inc., 420 Lexington Ave., New York. Price $455 yearly sub- 
scription, Includes weekly bulletin on recent developments In Washington, 

D.C. in legislation relating to education and research. 

This Is an expensive reference source, but the services provided 
to the subscriber are well worth the Investment. Two large loose-leaf books 
are. .Irxluded, containing detailed and very much up-to-date Information on all 
major developments In the field of education. The volumes are revised each 
week .v'hen the company (CC) malls supplementary loose-leaf pages to all su!i • 
scrloers. Included, too, are weekly bulletins dealing with recent davelcp- 
cients Iti Wsohlngton, D.C., and a copy of each law or pending law In both the 
House «id Senate pertaining to education. This is a must for larger reseat :;i 
organisation. 

Programs and Services; U.S. Dent, of Health. Education and Welfare . 

Supt. of Documents, U. S. Printing Office, Washington, D.C., 20402, Price $2.00. 

This Is a useful and comprehensive guide to federal prograns admini- 
stered through the Dept, of Health Education and Welfare. In light of t\i<i 
coverage and relative cost, this is highly recommended as a valuable refer 
source. 

of Education Support for Research and Related Activities . 

Dept. HEW. Washington, D.C. Free (?) on request. 

This 22-page manual provides sunanary information on pattern*? of 
support and application procedures througfi the U.S. Office of EducatiL-xi. It 
Is a helpful reference guide for anyone Involved In educational research. 

5. G rant Data Quarterly (1st four Iss. ^s in 1967). Academic Media, Inc. 

10C35 Santa Monica Blvd., Los Angeles, California 90025. Price: Single 

subscription $35.00 (10% off on 2 or more). 

The first four Issues present detailed i.-;foriaatio*i oa govevnrent 
support programs, buflneos and professional organization support programs, 
and foundation support programs. This quarterly would be valuable as a ref- 
erence source for college libraries, or progressive departments contemplate 
a substantial volume of research and development activities. 



o 
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Appendix D 

Partial List of Fund Sources for Educational Research 



A, Cooperative Research Program (HEW) 

• References ; See College and University Reporter (C»C.. Inc,), 
at 1651; also 8911 and 8401. See Programs and 
Services of the U. Department , HEW, pp, 223—224, 
and 231-232. ^ 

1. The Cooperative Research Programs^ administered by 
the U. S. Office of Education include support for both 
basic and applied research, demonstration projects, 
and curriculum improvement projects. Funds were 

also authorized for the creation of research and 
development centers, educational laboratories, and 
for various developmental activities, 

2. Small contracts . Perhaps of greatest significance 
to the researcher just beginning a career in educa- 
tional or behavioral research is the small contracts 
program. This program is intended to provide support 
for small-scale research or development projects 
which require less than $10,000 in federal assistance. 

The program "supports experimental research, surveys, 
demonstrations and curriculum studies," and also 
"assists in making exploratory studies designed to 
determine the feasibility of more extensive research 
on specific problems." 

Proposals are submitted through the Bureau of 
Research, 3 U. S. Office of Education, HEW. There are 
no specific deadlines for small contract proposals, 
and the prograij is designed so that proposals are 



^ Authorized by the Cooperative Research Act of 1954, with 
extension through the Elementary and Secondary Education Act 
of 1965. 

2 

College and Univ . Reporter (1651) 

^ Through the regional offices 
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processed with a minimum of delay. The reader should 
refer to the Office of Education pamphlet "Support 
for Research and Related Activities” for details on 
proposal format. 

3. See references above for discussion of large-scale 
project or program support under the Cooperative 
Research Act. 



B . Rc jearch in Educational Media (HEW) 



References: See College and University Reporter (C.C.y Ire.) 

at 1653-1654, 8671 and 10,001. Also for PL 89-209, 
see 9651. 

1. Title VII of the National Defense Education Act of 1958 
(PL 85-864 as amended) provides support for research 
and experimentation in more effective utilization of 
educational media. ^ This includes television, radio, 
motion picture films, slides, tapes, programmed instruc- 
tional devices, and other media designed to supplement 
instruction. Part A of Title VII provides support for 
research, while Part B includes authorization for dissem- 
ination of media (or Information on application of new 
media) . 



2. This program, also administered by the Bureau of Research, 
provides for several types of grants -- both large and 
small, and has wide applicability across subject-^tter 
areas. It is possible to obtain support of $10,000 or 
leas for projects designed to improve or evaluate media 
applications within a department or specific course U.e., 
minimum generaUzability) . Small grmt proposals are sub- 
mitted to Office of Education .regional offices. 



C. National Science Foundation (NSF) 

References: College and Oniveraity Reporter. 4568-4573; 

end 9001-9051. 



^ The National Foundation on Arts and Humanities Act of 1965 
(PL-89-209) Provides authorization for research on media in 
arts and humanities instruction. 
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!• The Social Science Division of the National Science 
Foundation provides support for basic research, for re- 
search in the anthropological sciences, econotaic sciences, 
sociological sciences, and research in the history and 
philosophy of science. Authorization is through the 
National Science Foundation Act of 1950 (PL 507) . 

2. Specific programs among those alluded to above require 
different guidelines, and each program includes individ- 
ual deadlines for submitting proposals. Be sure to note 
deadlines in planning a proposal. You shouH count on 
several months of thought, writing, and criticism from 
colleagues in preparing a research proposal. This is 
extremely important.^ 

3. One recent emphasis within NSF has been on science educa- 

tion, and there are now three divisions within the founda- 
tion which are responsible for the various science pro- 
grams in education: Division of Pre-college Education, 

Division of Undergraduate Education, and the Graduate Ed- 
ucation Division. Each division is responsible for a 
variety of science education programs, many of which over- 
lap those of the U.S. Office of Education. 

Arts and Humanities Endowment Funds 

References : College and University Reporter. 1720, 1722, 

1724, 9651. 

1. Research support is available for both broad .and specific 
studies, with emphasis upon American histox^ and literature. 

2. The Humanities Endowment cooperates with the Office of Ed- 
ucation in sponsoring research on teaching of the humani- 
ties at the pre-college level. 

i 

3. Send for paoq>hlet: National Endowmeut for the Humanitir>r - 
Initial Programs . September 1966, for additional isf;/rma- 
tion, or refer to source listed above. 



^ NSF proposal guidelines may be found in College and Univer- 
sity Reporter , beginning at 5401, 



IX- 36 



E • Other Sources* 

1. Office of Naval Research (D09) 

2. Department of the Army (DOD) 

3. Advanced Research Project Agency (DOD) 

4* Agency for International Development 

*See Guide to Support Programs for Education . 3M Company, 
St. Paul, Minn., 1967 



*See also pgs. 16 and 17 of the USOE pamphlet (Support for 
Research and Related Activities) for programs administered 
by the Bureau of Research. 

National Institut e of Mental Health ^ <N1MB) 



**R aference3 ; College and University Reporter . 4125 
through 4128. 



1. The NIMH supports basic and clinical research relating to 
the etiology, diagnosis, treatment, and prevention of 
mental illness . Grants are also available for Instruc-' 
tional research and teaching. 

2. NIMH further provides funds in support of research on 
mental retardation . Support is available on the treat- 
ment, care, management and training of mentally retarded 
in light of biological, psychological, or sociocultural 
factors Involved. 



**Future emphasis will be on ^’research on child rearing 

practices, teaching programs, parent and child therapy 
techniques, and interdisciplinary approaches to treat- 
ment and rehabilitation.” ' 



Dlvsion of the Public Health Service, HEW. 
^ College and University Reporter . 4126 . 
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Child Health and Human Development 

Reference s; College and University Reporter , 4111 

1. The national Institute of Child Health and Human 
Development - a division of the National Institute 
of Health (NIH) of the Public Health Service 
(Department of Health, Education and Welfare) - provides 
support for research and training related to "aacernal 
health, prenatal care, child health, and human develop<- 
ment.'* NIH emphasizes research In four areas: 
reproduction, growth and development, mental retarda- 
tion, and aging. 

2. Specific guidelines and notification of deadlines for 
proposals should be requested from the National 
Institute of Child Health, Public Health Service. 
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Utilizing Library Resources In a Search of Literature 



The first iter to remember irtien beginning a search through the 
literature Is that the reference librarian Is a major source of Information* 
The reference librarian can save countless hours « if only the researcher 
will present the librarian \;ith Infonnatlonj specific if possible « on what 
is being sou^t. The reference libra tian can quickly point to the most 
pertinent information* Crucial books may be kept at the reference desk, 
so don*t overlook asking for suggestions about books whlbh may be held be- 
hind the desk* For example, the book How to Locate Educational Information 
pad Data is a reference source which is usually kept at the librarians desk* 

The card catalog Itself is a major resource* Most large libraries 
have extensive subjects headings, in red, for subjects pertinent to re- 
search* If "educational research" Is not located under that heading, try 
"Research, Educational"* Books Which contain summaries of educational re- 
search may be located through the card catalog* Also« large libraries 
have special librarians who are authorities in their field, l*e*, social 
science, education, etc* This specialist can be called upon if the 
reference librarian is not aware of some of the resources* 

Indexes) Ilajor sources of information for educational researcherr< 

are: 

P-Sycholo^^ Abstracts (indexes of authors and subjects at 
the end of the volume; also separate bolumes containing 
Indexes to authors and subjects, cumulative for the past 
several years, are nox 7 available); 

(2) Education In^ex* ^^lasslfled by author, title and subject 
(remember that in most indexes author and subject are 
usually more accurately Included than are titles); 

(3) Sociology Abstracts (have separate Indexes, of author and 
subject); 

(4) Dissertation Abstracts (has a cumulative index for the year) ; 

(5) There are technical indoxes sv.ch as Opdrationa Besearc?'' / 
Ifanagement Science^ U*S« Government SeseArch and Develo o- 
ment reports (prior to 1963 Indexing is terrible, almost 
impossible to find anything except under subject) ; 

(6) Business and Periodical Index 4 which contalim references 
to articles dealing tTlth business education, for example* 

Books in Print* Subject Index; and, 

(8) American Library Assoclat^Jon, Standard Catalog. 

(9) Become acquainted with the E«R«I*G* System* This 
Educational Research Information Center, established by the 
U*S*0*E. Bureau of Research, appears to be developing into 
the most useful and comprehensive source* See in Section X, 
the Support for Research booklet, p* 5* 
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Section X 
USOE Support 



T(70 u 8 e£tiX documents frwn the U»S» Office of Education 
are contained In this section. 

One describes the activities supported by the Bureau of 
Research. The appendices listing Regional Offices and ERIC 
Clearinghouses may be of particular value in your initial re- 
search effects. 

The other describes the Small Project Research Program. 
This program Is highly relevant to the background and Interests 
of participants in this Institute. The proposal format and 
application procedure are Included. 
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Foreword 

SYSTEMATIC RESEARCH, DEVELOPMENT, AND DISSEMINATION are Indispen- 
sable to the improvement of education. They are the means by which the 
educational enterprise assesses its current strengths and shortcomings, works 
out new programs and techniques to meet changing needs, and provides 
objective informat'ion and models on which the schools and the public can 
base their individual steps toward educational improvement. 

Through support of research and related activities conducted outside the 
Office of Education, the Bureau of Research encourages efforts to develop and 
provide the kind of education today’s society needs and tomorrow’s society 
has every right to demand. Research support complements other forms of 
support, whether from local. State, private, or Federal sources. It provides 
useful blueprints of promising alternatives and thus gives direction and 
momentum to education’s forward thrusts. 

This booklet describes the wide range of activities which receive support 
administered by the Bureau of Research and the general application procedure 
to be used by those who seek support under any of the Bureau’s authorizations. 



R. Louis Bright 

Associate Commissioner for Research 
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INTRODUCTION 



THE OFFICE OF EDUCATION’S BUREAU OF RE- 
SEARCH se^ks to improve education through 
support of a variety of research and related 
activities. Through authorizations from the Con- 
gress, the Bureau provides funds for research 
projects and programs designed to expand knowl- 
edge about the educational process, to develop 
new and improved educational programs and 
techniques, to disseminate the results of these 
efforts to educators and the public, and to train 
researchers in the field of education. This booklet 
presents an overview of the kinds of activities 
^ administered by the Bureau. Its appendixes con- 
R ^in general application instructions (including 
criteria for selection of proposals for funding), 
Jj summaries of pertinent legislative authorizations, 
and other information of interest to applicants for 
support. 
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One major factor distinguishes activities eligible 
for support through this Bureau: They must be 

D research or research related, as distinguished 
from operational activities which receive support 
through other Office of Education bureaus.^ Re- 
search, development, demonstration, and dis- 
semination help provide objective bases for the 
improvement of operational programs at all levels 
of American education. 

The Bureau of Research awards support under a 
number of legislative authorizations, the most 
flexible of which is the Cooperative Research Act, 
as amended by Title IV, P.L. 89-10. Under this 
act, it is possible to award grants or contracts 
for any research or related activities which 
promise to benefit education. Other authorizations 
permit support of research and related activities 
in a variety of categorical areas: Educational re- 
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search in foreign countries; research and develop- 
ment relating to foreign language teaching and 
language development; research on educational 
uses of new communication media and dissemina- 
tion of the resulting information; research and 
demonstrations dealing with the education of 
handicapped children and youth; research, devel- 
opment, and training in vocational and technical 
education; and library and information science 
research. (See appendix B for information on 
legislative authorizations administered by the 
Bureau.) 



Concentration of administrative responsibility for 
all these authorizations within a single Bureau 
simplifies application procedures and facilitates 
coordination among programs. Bureau staff re- 
ceive all incoming proposals, analyze them to de- 
termine whether they are suitable for Bureau 
support, and allocate them for necessary internal 
and external review. The Bureau has five divi- 
sions: Adult and Vocational Research, Elementary- 
Secondary Research, Higher Education Research, 
Research Training and Dissemination, and Labora- 
tories and Research Development. There are also 
separate staffs for the Arts and Humanities Pro- 
gram and for the Regional Research Program in 
the Office of the Associate Commissioner. Re- 
gional offices are being staffed to administer 
Small Project Research and Research Develop- 
ment Grants for small and developing colleges. 
(See appendix C for information on the nine OE 
regional offices.) 



* For information about Office of Education support for non- 
research activities, inquiries should be sent to the Bureau of 
Elementary and Secondary Education, the Bureau of Higher 
Education, or the Bureau of Adult and Vocational Education, 
whichever Is applicable. 
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There are two patterns of research support avail- 
able from the Bureau of Research: Project sup- 
port and program support. 

Application for project support should be made 
v/hen the initiator wishes to engage in a self- 
contained activity or set of activities with a few 
well-defined objectives that can be carried out in 
a predetermined period of time. A great many 
different kinds of projects can be supported 
simultaneously under this pattern. Program sup- 
port is applicable for specifically announced 
problem areas in education where the Office of 
Education feels there is a need for continuous, 
intensive attention. Programs provide for a con- 
centration of professional resources on these 
areas over an extended period of time. Thus, 
project support deals with clearly delineated, 
limited-time research (of any magnitude), while 
pr ••'am support is reserved for certain groups of 
continuous research or re search -related activities 
which are able to adapt to evolving situations. 

Projects are usually planned and initiated by those 
who submit them as proposals. The relatively 
flexible format of the General Application Instruc- 
tions, appendix A, frees the applicant to focus 
upon the activity or group of activities he pro- 
poses to carry out. These instructions should be 
sufficient for applying for support for most proj- 
ects. However, those seeking program support 
may need to request special sets of instructions 
that supplement the material in this booklet. 

Project Support 

By far the greatest number of activities admin- 
istered by the Bureau receive project support. 
Subjects of project research are as varied as the 
questions educators seek to answer. For example, 
some projects may explore educational needs or 
seek to resolve issues. Others may develop ma- 
terials or methods, or test them in controlled situ- 
ations or field studies. Still others may investigate 
the potential of promising programs or practices 
to bring about desirable educational change — in 
fact, they may investigate factors related to the 
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change process itself- Projects may analyze, con- 
solidate, or synthesize information from research 
or from practice. They may demonstrate or dis- 
seminate educational information or techniques. 

Some projects may seek to advance knowledge or 
solve theoretical problems in the behavioral 
sciences. Others may involve applied research or 
curriculum development in specific content areas 
or combinations of areas. Still others may pro- 
duce materials for educational technology or 
demonstrate their uses. Projects may be directed 
toward any educational level or toward students 
with particular problems or characteristics, in 
short, the Bureau adjusts its support to whatever 
kinds of projects promise effective and practicable 
innovations. In this way, it provides freedom for 
a viable, self-renewing approach to educational 
improvement. 

Project Research in General . — ^There is no limita- 
tion on the size, the area of study, or the kind of 
activity or activities eligible for project support — 
so long as they deal with educational resear 
development, or dissemination. However, t 
public interest and the critical needs of education || 
demand careful administration of available funds. » 
Sometimes this means judicious selection from 
among similar proposals to avoid unnecessary n 
duplication. Sometimes it means making support || 
of a promising research activity contingent upon 
the investigator’s ability to redesign his research » 
so that its results will be useful in settings other M 
than his own. The size of a study is no measure 
of its potential for educational improvement. In 
some cases, a study must be quite large in order || 
to produce valid information; in other cases, a " 
relatively inexpensive pilot study may accomplish 
a great deal. When it is necessary in terms of H 
total research goals, the Bureau may solicit pro- 9 
posals in critical areas where research interest 
has been slow to develop. 



All proposals are assessed in terms of their 
promise for meeting stated objectives, the signifi- 
cance of those objectives to the Office of Educa- 
tion’s total effort, and economic efficiency. Small- 
scale research or development projects can be 
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funded with a minimum of delay for proposal re* 
■ view. Consideration of larger projects takes 
I longer. 
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S,nall Project Research refers to those activities 
which require no more than $10,000 from the 
Office of Education and take no longer than 18 
months for completion. This special classification 
has been provided to give adequate consideration 
to inexpensive yet worthwhile projects; to en- 
courage personnel of small colleges to gain ex- 
perience in research and related activities; and to 
support significant small-scale educational re- 
search projects by doctoral and postdoctoral 
students and fellows, particularly those at devel- 
oping institutions. 

Support for these activities is being decentralized 
to the OE regional offices. This method of ad- 
ministration should facilitate prompt considera- 
tion of proposals and bring evaluation, negotia- 
tion, monitoring, and other assistance closer to 
those carrying out the projects. The instructions 
appendix A are generally applicable to small 
'ojects, but separate instructions are available 
by writing to the regional offices or to the Regional 
Research Program staff in the Bureau. 



Program Support 

y In addition to the wide array of projects described 
above, the Bureau of Research also supports con- 
tinuous programs, each of which provides a par- 
ticular thrust in the total research and develop- 
ment effort. This type of support Is used for the 
activities of research and development centers 

D and educational laboratories, for training educa- 
tional researchers, and In other appropriate cir- 
cumstances. These programs allow long-term 

D staff commitments and continuous development 
and adaptability. Appropriate supplementary in- 
structions for applying for program support are 

□ available from the Bureau of Research. (See 
programs below.) 

Research Development Grants support the efforts 
of small or developing colleges to acquire sound 
research orientations. Their personnel may use 
the funds to develop their own research skills, to 
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research their own educational problems, or to 
try out premising educational innovations. A 
primary purpose is to help teachers and future 
teachers learn to use research, research results, 
and the research or inquiry approach In their 
classrooms. 

To encourage the development of research capa- 
bilities in smaller Institutions of higher education, 
especially those which train teachers, grants are 
available to consortia or groups of institutions 
which plan to combine their ideas and competen- 
cies in a research development activity. Single 
institutions may also apply but should scale their 
programs and fund requests appropriately. Partic- 
ipating institutions should expect to (1) dissemi- 
nate information on research findings and re- 
search design, administration, and management; 
and (2) provide their staffs with time and money 
to conduct the research activities. 

CE regional offices are being staffed to administer 
Research Development Grants. Separate instruc- 
tions with examples of appropriate activities are 
available either from the Bur-eau of Research or 
from the regional offices. 

Research and Development (R & D) Centers. — A 
research and development center concentrates on 
a single problem area in education and conducts 
activities ranging from basic research through dis- 
semination. Centers are usually established at 
universities or other institutions where staff have 
already demonstrated exceptional competency in 
the particular problem area and can be expected 
to produce early, continuous, and significant edu- 
cational advances. The sponsoring institution 
generally continues to provide s . nntiai local 
support for center activities. 

Each center is interdisciplinary and ordinarily 
maintains cooperative relatic'rships with regional 
laboratories. State departments of education, 
local school systems, universities and teacher 
training colleges, and relevant professional and 
nonprofit organizations. Within its established 
area of investigation, each center can direct its 
own program without obtaining prior Office ap- 
proval for individual projects. Thus, center ac- 
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tivities can reinforce each other, promising leads 
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and research findings can quickly be put into 
practice. 



At the end of fiscal 1966, 11 of these centers 
were in operation, each working in a major prob- 
lem area. The comprehensiveness of center op- 
erations and the cost of staff dictate extremely 
careful evaluation of proposals for establishment 
of new centers. Care must bs taken to select only 
problem areas which need continuing investiga- 
tion. Each center must have the professional and 
administrative competencies to direct activities 
which constantly evolve from these Investigations. 

Those persons seriously planning to apply for 
support to establish an R & D center should con- 
sult with Bureau staff of the R & D Center Pro- 
gram and secure supplemental instructions before 
drawing up a proposal. These instructions explain 
the special conditions applicable to support for 
R & D centers, including site visits, monitoring, 
and arrangements for continuation funding. Indi- 
viduals interested in working with one of the 
centers should apply directly to the center, and 
not to the Bureau of Research. A list of R & D 
centers, their locations, and their areas of inquiry 
is provided in appendix D. 



Educdtiondl Laboratories . — Educational labora- 
tories differ from R & D centers In their focus, 
composition, and activities. Although both may 
work all along the continuum from basic research 
to dissemination and Implementation, centers 
emphasize research and development while labo- 
ratories stress development, dissemination, and 
implementation. Each laboratory is primarily con- 
cerned with educational improvement in a partic- 
ular region, especially with wider adoption of 
beneficial educational innovations there. To this 
end, the laboratoi^ designs its own program and 
continuously adjusts it to meet emerging needs 
of the region. In terms of organizational struc- 
ture, the laboratories are new institutions which 
draw upon colleges, universities, State educational 
agencies, local schools, private industry, and 
other educational interests for their staff, mem- 
bership, and affiliations. 






Laboratories are set up through the initiative of 
individuals and groups in the regions. These ffl 
local and regional efforts have resulted in 20 SI 
laboratories, generally organized as nonprofit 
corporations, which form a network sen/ing all of m 
the continental United States. Individuals and g 
groups interested in effecting educational im- 
provements through the regional laboratory pro- 
gram should contact the laboratories directly, g 
not the Bureau of Research. A list of the 20 " 
educational laboratories with their addresses and 
participating States is provided in appendix E. 



Programs for Training Educational Researchers. 
— ^To provide for sound educational research and 
development in the future, support is available 
for institutions to train researchers and to develop 
and improve their own programs for such training. 
Institutions may request support for under- 
graduate, graduate, and postdoctoral training; 
and for training institutes, inservice programs, or 
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Funds may be used to develop and strengthe 



special projects dealing with educational research. 9 

research training staffs and curricular capabilitieSr 
and for stipends and institutional allowances for 
trainees. Training may be concerned not only 
with research per se but also with educational 
strategies needed to bridge the gap between 
research and practice. 
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Ability to provide such training, potential for pro- 
gram development, and equitable geographic dis- 
tribution are considered in the establishment of 
these programs. Cooperative programs among 
sponsoring Institutions and with State agencies, 
regional laboratories, and other educational 
groups are encouraged. Special instructions and 
application forms are available from the Bureau’s 
Division of Research Training and Dissemination. 9 
Individuals wishing to participate in these « 
research training programs should apply to the 
institutions which provide the programs, not to m 
the Bureau of Research. || 

Through a limited number of National Post- 
doctoral Fellowships in Educational Research, the 9 
Office of Education arranges with institutions to 9 
provide a year of intensive research experience 
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for selected candidates. Further information on 

a Postdoctoral Fellowships is available from the 
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semination. 
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Educational Research Information Center (ERIC). 
— ERIC is a comprehensive national information 
system designed to serve American education by 
making available reliable, current educational re- 
search and research-related materials. The sys- 
tem is made up of a network of information 
I clearinghouses or documentation centers located 
I throughout the couniry and coordinated through 
Central ERIC in the Office of Education. 

I By the end of 1966, clearinghouses had been 
■ established in 13 substantive areas (see appendix 
F). As funds become available, the Bureau will 

S issue requests for proposals to establish addi- 
tional clearinghouses. Each clearinghouse collects 
materials in a different subject area and is staffed 
IB by specialists who are responsible for document 
^analysis, selection, and other activities related to 
ffl^lC’s mission. Documents collected throughout 
^e ERIC system are abstracted, indexed, and put 
S on microfiche, 4" x 6" film cards that contain up 
" to 60 pages of text per card. 

Starting in November 1966, a monthly journal, 
RESEARCH IN EDUCATION, is being issued by the 
Office of Education through the Bureau of Re- 
search, Division of Research Training and Oissem- 
ination.2 Each issue contains bibliographic 
citations and abstracts of recently funded projects 
and of final reports of completed projects sup- 
ported through the Bureau of Research, as well 
as detailed indexes of cited research documents. 
Subsequent issues will Include documents from 
the 13 ERIC clearinghouses. A cumulative index 
will be published annually. 

Other , Specialized Programs. — In addition to the 

- Subscriptior.s ($11 domestic, $13.75 foreign, $1 single copy, 
domestic) are available from the Superintendent of 
U.S Government Printing Office, Washington D.C. 20402. All 
reports cited In RESEARCH IN EDUCATION are available as 
hard copy or microfiche from the ERIC Document 
Service, opeiated by Bell and Howell Company, 1700 Shaw 
Avenue, Cleveland, Ohio 44112, under an OE contract. Prices 
ar'e quoted 3s part of each citation. 
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relatively general programs described above, the 
Bureau of Research also administers support for 

’fnllr\ui5ncf cr>or'ia!iTorl nrocn^5»rr|Q 

Training of Personnel >n Vocational Education may 
be provided through Bureau-supported institutes 
to prepare teachers for new and changing occupa- 
tions and to upgrade the competencies of teachers 
already engaged in adult and vocational programs. 
Participants may include not only vocational 
teachers and counselors but also school adminis- 
trators and related vocational-technical-adult 
education personnel. Support may also be 
requested for seminars to improve competencies 
of individuals concerned with vocationai educa- 
tion research and development activities. Forms 
and instructions for making application are avail 
able from the Division of Adult and Vocational 
Research, Bureau of Research. 

State Research Coordinating Units in Vocational 
Education, located In all but a few States, coordi- 
nate local research and demonstration efforts in 
vocational education and provide consultative 
assistance to individuals who wish to prepare 
research proposals in that field. Support admin- 
istered through the Division of Adult and Voca- 
tional Research is gradually being reduced as 
States assume this responsibility. 

Instructional Material Centers for Handicapped 
Children and Youth are being established to pro- 
vide a network of repositories for such aids as 
braille books, test kits, and tapes and recording 
devices where staff can evaluate their effective- 
ness and make them available to local schools. 
The centers will also engage in research and 
development aimed at improving teaching mate- 
rials for the handicapped and hold institutes and 
workshops to familiarize teachers with the use of 
special educational materials. Appendix G lists 
the 10 centers established by the middle of 1966. 
As funds become available, new centers may be 
added to serve particular geographic areas. Infor- 
mation is available from the Handicapped 
Children and Youth Research Branch, Division 
of Elementary-Secondary Research, Bureau of 
Research. 
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APPLICATION PROCEDURES 

The Official Application 

Application for support is made by submitting a 
standard formal proposal, accompanied by an 
offidai application form. This proposal Is eval- 
uated for its merit by the Office of Education staff 
and by non-Federal field readers selected for 
their research or specialized experience and gen- 
eral knowledge of the field. It Is the only contact 
between the reviewers and the Initiator’s Idea; If 
it does not convey the message, the staff and 
readers will not assume meaning or intent. If the 
activity is approved for funding, the proposal 
document becomes part of the contract. 

It is suggested that the Initiator ask several per- 
sons who are not close to the problem to read the 
proposal to make certain that It communicates 
clearly. Many potentially good research proposals 
fall short of recommendation for negotiation 
because the procedures are not clearly written 
or omit essential details. Others present so many 
questions for research that the study would be 
unwieldy or could not be completed with the 
funds and within the time requested. Should the 
proposal fail to be selected for funding, a sum- 
mary of the reviewers’ comments may be obtained 
upon request and a revised proposal may be sub- 
mitted. However, reconsideration by the Bureau 
is no guarantee of subsequent selection. 

Preliminary Review 

Before preparing a formal proposal for either 
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project or program support, an individual or group ~ 
may find it helpful to solicit suggestlon.<i from the || 
Bureau by sending three copie? of a brief (not 
over five pages) description of the activities to 
be undertaken to determine whether they would ^ 
be eligible or appropriate for consic’sratlon under 
Bureau «iuthorlzations. The description should 
include ai title, a list of objectives or questions 
which the research would answer, a brief descrip- 
tion of the procedures, and an indication of the 
significance of the activity for educational im- 
provement beyond the setting In which It would be 
carried out. The first page should be clearly 
marked ‘’Preliminary Statement” and should 
include the applicant’s full name, his professional 
title and the organization with which he is affil- 
iated, and his telephone number. Including the 
area code. The preliminary statement should be 
sent to the mailing address for formal proposals 
(see page 12). 

This preliminary review Is Intended to help the 9 
potential investigator avoid spending time on 
formal proposal that would be Ineligible S0 
Inappropriate. However, suggestions from the 
Bureau should not be Interpreted as either deny- 
ing the proposer the right to submit a formal 
proposal or as guaranteeing approval. Since 
Bureau staff are obliged first to give attention to 9 
review of formal proposals and to monitoring of 
approved projects, preliminary drafts are reviewed 
as time permits. The potential applicant, there- 
fore, Is encouraged to forward his prospectus well 
In advance of the time he plans to submit a 
formal proposal. 
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APPENDIX A. GENERAL APPLICATION INSTRUCTIONS 



Activities Appropriate for Support 

To be eiigibie for support administered by the 
Bureau of Research, an activity must (1) be 
research or research-related, (2) show promise of 
improving education, (3) have general (not purely 
local) applicability, and (4) be directed toward 
communicable results. 

The term “research related" is broadly inter- 
preted to include such activities as the develop- 
ment of materials and improvement of instruc- 
tional practices in general and specific areas, 
and dissemination and implementation of the 
results of research. Local projects must lead to 
findings significant for other settings if they are 
to be considered eligible for support under 
research authorizations. This applicability or 
transferability to other settings is the factor 
which, in many instances, determines whether a 
given proposal is eligible for consideration by the 
Bureau of Research. 

The Bureau’s authorizations do not include 
funds for general operating programs or for activi- 
ties carried on in conjunction with the research 
which are part of the normal activities of project 
personnel (e.g., teacher salaries in experimental 
classes). Bureau support may be requested for 
the research component of larger projects where 
the operating costs are funded from other 
sources. In a few instances, support may be pro- 
vided for those aspects of an operational program 
which must be modified or established in the 
interests of the project. 

*Vith the excciption of Small Project Research 
and Research Development Grant activities, there 
are no established time or cost limitations for 
projects which may be considered for support. 
However, contractual arrangements usually obli- 
gate project funds only out of the current year’s 
appropriations. If a proposed activity involves 
extensive costs or must be continued over sub- 
sequent years, the proposal must undergo sub- 
stantial additional evaluation to assure sound 
initial investment. 



Eligible Parties 

A record of noncompiiance, by an applicant, with 
the terms of prior research grants or contracts 
(for example, failure to submit a final report) may 
be sufficient grounds to refuse consideration of a 
proposal. 

Grants or contracts may be awarded to colleges, 
universities. State departments of education, or 
to other public or private agencies, organizations, 
groups, or individuals after proposals have been 
reviewed by Office of Education staff and by 
appropriate non-Government advisory personnel 
and approved by the U.S. Commissioner of Edu- 
cation. With profit-making organizations, contracts 
follow Federal Procurement Regulations. 

The type of support instrument — i.e., grant, cost- 
reimbursable contract, or fixed price contract — 
may be selected by contract personnel during 
negotiation. If the authorization providing the 
funds does not specify wliich type of support 
consideration is given to providing the be^ 
balance between fiscal control and flexibility of 
operation. All research grants must include 
significant cost sharing by the grantee institution. 
Moreover, in some contract research activities, 
non-Federal contributions are appropriate and a 
factor in award consideration. 

Eligibility for larger projects and for the beginning 
phase of longer projects may be limited to groups 
which are or can be organized so as to ensure 
proper continued professional and fiscal account- 
ability and competence. 

Public schools, school districts, and local edu- 
cational agencies, in order to assure appropriate 
liaison and communication with operating school 
systems, are requested to send three copies of 
their proposals to their State department of edu- 
cation for comment. To be eligible for support 
under the Vocational Education Act of 1963, pro- 
posals submitted by local educational agencies 
must contain evidence of approval by an author- 
ized official of the State board of vocational 
education. 
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|! II Criteria for Evaluation of Proposals 
II * * Proposals are evaluated and recommended for 
approval according to these criteria: 

1. Educational significance. 

2. Soundness of design, procedure, or 
plan. 

3. Adequacy of personnel and facilities. 

4. Economic efficiency. 

5. Other specific criteria, as appropriate. 
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Evaluation of a proposal’s significance to educa- 
tion requires more than consideration of the proj- 
ect itself. It involves attention to the breadth of 
the project’s probable impact, its relationship to 
other ongoing and completed research, and its 
capacity for contributing to educational improve- 
ment within the context of total research needs. 

Educational significance, then, is the first of 
^ vlveral conditions for support; but if a proposal 
^ lacks adequate technical quality, personnel, or 
economic efficiency, it will not be supported, 
regardless of its significance. In like manner, no 
matter how technically excellent the proposal may 
be, its support is contingent on its significance to 
educatiori nationally. 

Funds are not available to support all the good 
proposals submitted to the Bureau of Research; 
the current approval rate is about one in five. To 
provide a balanced effort toward educational 
improvement within the context of active com- 
petition for available Federal research funds, it is 
sometimes necessary to forego support in a well- 
researched area In order to attend to a neglected 
but critical one. There also are instances when 
difficult choices must be made between one or 
two relatively large projects and a larger number 
of smaller ones. In any case, selection of any 
given activity for support is based on systematic 
evaluation of the plan set forth in the formal 
proposal document and determination of whether 
it meets the needs of education. 



The Proposal Document 

Application for support is made by submitting a 
properly executed standard application form 
(available from the Bureau of Research or 
regional offices) and the appropriate number of 
copies of a formal proposal which describes the 
activity or activities, explains their significance, 
identifies key personnel, and estimates costs. 

Since funds are available only for the most 
promising activities, proposals must be assessed 
competitively (see Criteria for Evaluation of Pro- 
posals above). A uniform proposal format has been 
designed to accommodate all Bureau of Research 
projects, regardless of the authorizations under 
which they are funded. Use of a single format 
gives the applicant freedom to concentrate on the 
particular activity to be undertaken without having 
to choose among application patterns. 

All proposals must include the standard cover 
page, a one-page abstract, the body section, and 
personnel and budget items. Within this frame- 
work, each applicant states the case for his 
activity. 

The guidelines below are generally applicable, 
regardless of the magnitude of the proposed proj- 
ect, the area of investigation, or the authorization 
under which the project may be funded. If 
approved. The applicant is expected to make 
judicious adaptations of this format to accom- 
modate the kind of research or related activity he 
proposes to undertake. 

L The Cover Page . — Nothing may precede this 
page. It contains only the information on page 
10 in the order indicated, and two copies must be 
signed by the Initiator or Project Director and by 
the Transmitter. 

II. The Abstract occupies a single page. Identifies 
the proposal, and concisely and simply .sum- 
marizes the contents. To accommodate the vari- 
ous uses made of this page, the abstract must be 
written in language understandable by an 
Informed layman. One abstract is placed after 
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PROPOSAL FOR RESEARCH AND/OR RELATED ACTIVITIES 
SUBMITTED TO THE U.S. COMMISSIONER OF EDUCATION FOR 
SUPPORT THROUGH AUTHORIZATION OF THE BUREAU OF RESEARCH 



Title: 


(Be concise; avoid obscure technical terms) 


Applicant Organization: 


(Name of College or University, State Agency, 
School District, or other unit with major com- 
mitment to the activity)* 


Initiator: 

(Principal Investigator) 


Signature 

(Full name of person responsible for development 
of the proposal; position; telephone area code, 
number, extension) 



(If initipted by ^ student under faculty sponsorship, add line — sponsored by 

3l^natUr© cn\/0 com a I 



sponsor,) 




Project Director: 

(if different from Initiator) 


Signature 

(Full name of person in charge of investigation; 
position; telephone area code, number, extension) 


Transmitted By: 


Signature 

(Full name and position of official committing the 
applicant organization or cooperating institution, 
- jency, school district, etc., to the activity; tele- 
phone area code, number, extension) 


Duration of Activity: 


(Proposed beginning and ending dates) 



Total Federal Funds Requested: 



Date Transmitted: 

* 2^ private organizations should adapt the format of this cover page to show information comparable to 

that required of others. 
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the title page in each copy of the formal proposal, 
and 20 copies are provided separately. At the top 
of the abstract page, the following items should 
appear on separate lines: Title of Project, Princi- 
pal Investigator, Contracting Agency, Amount of 
Federal Funds Requested, and Proposed Begin- 
ning and Ending Dates. The summary portion of 
the abstract has three parts: A statement of the 
purposes, objectives, or nature of the project; an 
ind’- ;ition of the expected contribution to educa- 
tion; and a compendium of procedures or descrip- 
tion of what is to be done. 

III. The Body of the proposal communicates the 
project director’s plan and its probable effective- 
ness. It should be clear, concise, forthright, and 
complete, and should be kept within 20 single- 
spaced, typewritten pages. Proposals for excep- 
tionally extensive or complicated projects may 
subsequently be developed into larger documents 
with the advice and cooperation of Office staff. 
The body of Small Project Research proposals 
should not exceed 10 pages. 

within the body section, there are normally three 
parts which should be subtitled as appropriate 
for the particular kind of research, development, 
or dissemination activity to be undertaken. These 
parts answer the questions: Why? What, when, 
and how? and Use of results? Some attention is 
also given to personnel, facilities, and costs, 
which are subsequently treated in detail. 

A. Problem and Objectives. — ^The first part 
tells why the research or related activity should 
be undertaken. It includes a statement of the 
problem or purposes, review of literature and 
related research, concise statement of objectives, 
or any other information necessary to establish a 
sound rationale. 

B. Description of Activities (Procedures ). — 
This part tells what is to be done, when, and how. 
It Is the basis for determining the degree to which 
the proposed activity can be expected to accom- 
plish the objectives or satisfy the need set forth 
in the first part cf the proposal body. It delineates 
procedures, ou’dmes program arrangements, 



describes materials to be produced, or otherwise 
explains how the activity leads to results and 
how evaluation will be accomplished. Allowances 
for alternatives, if any, should be noted. (A time 
schedule for completion of the project is usually 
provided near the end of this part of the proposal.) 

The amount and kind of detailed information 
will vary according to the type of activity. For 
example, a basic research study — ^which might 
include rather strict statistical treatment — ^would 
be approached quite differently from a curriculum 
development activity. A project which is complete 
within itself would not be approached in the same 
way as a phase of a proposed multiphase activity. 
Preparation of initial materials for a modern for- 
eign language would not include the same pro- 
cedures as the demonstration of new media usage 
or the testing of a pilot study in vocational or 
technical education. 

C. The Use To Be Made of Findings . — ^This 
part tells how the results of the activity may be 
disseminated and/or implemented, what con- 
tribution to education can be expected, and what 
steps should follow. 

IV. Personnel and Facilities. — Personnel and 
facilities are a major determinant of capability. 

Personnel with major responsibilities are listed 
by name, position, title, experience, responsibili- 
ties within the project, percentage of time com- 
mitted to the activity, and the extent to which this 
commitment has been assured. Consultants who 
have agreed to serve should be similarly identified 
(otherwise, the application should describe the 
type of consultative assistance required). 

Facilities should be described, and the extent to 
which their use has been assured should be 
indicated. 

V. The Budget Section of the proposal starts on 
a new page and uses the tabular presentation 
shown at the end of appendix A. The applicant 
should show reasonable estimates but be detailed 
enough to suggest careful analysis of expected 
costs and understanding of fiscal responsibilities 



in connection with conducting the proposed activ- 
ity. AlternaHve budgets may show allowances for 
extension of objectives or revision of procedures. 
Local contribution, or cost sharing, is required on 
all grants, encouraged on contracts. Cost sharing 
may not be avoided by substitution of a contract 
in a situation requiring a grant instrument. Local 
contribution, while not necessarily a measure of 
the project’s soundness or potential worth and 
not specifically required on certain contracts, is 
considered evidence of local interest and commit- 
ment in the activity. The budget section should 
include an outline of the sources ancf amounts of 
non-Federal support and any conditions upon 
which this support is contingent. 

VI. Appended Items 

Other Information. — Give a brief statement about 
each of the following: (a) If this or a similar 
proposal has been submitted elsewhere, give 
details, (b) If this is a proposed extension of, or 
addition to, a previous or current project sup- 
ported by the Office of Education, give the 
Bureau and the grant or contract number of the 
related support instrument, (c) If there has been 
any previous communication with the Bureau of 
Research on this proposal, give name of staff 
member concerned. 

Revisions. — If this proposal is a resubmission of a 
previous formal proposal, give the Bureau pro- 
posal number assigned the original proposal and 
describe the major revisions which have been 
made. (This does not apply to preliminary state- 
ments submitted for informal review.) 

Report of Othrr Projects. — If any of the primary 
personnel have a current or uncompleted project 
with the Office of Education or other agency, an 
appended statement should indicate the status of 
the project, the amount of time devoted to it, and 
the relationships between the current and the 
proposed project. 

If any of the personnel have completed a research 
or development project supported by the Office of 
Education, give information to Identify it. If find- 
ings of the previous project are related to the 






current proposal, summarize them briefly. 

Agreement with Cooperating Agencies. — ^Where 
agreements with school districts or other cooper- 
ating agencies are a factor, copies should be 
appended. 

instruments. — ^When an applicant plans to use 
a published or unpublished instrument (e.g., a 
questionnaire or interview guide) in his study, the 
proposal document should include a copy or, if 
the instrument is still to be developed, a page of 
sample items and an outline of the complete 
instrument. (See Special Conditions, Data-Collec- 
tion instruments on page 14.) 

Other items. — If necessary, other items may be 
added, but appended items cannot carry the bur- 
den of the request for support. 
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Details About Submitting Proposals 

Forms for the standard application should be 
requested from the Bureau of Research at leas^ . 
2 weeks prior to the date the copies of the formaPi^ 
proposal are to be submitted. 
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Proposals should be on one side of standard 
( 8 V 2 " X 11 ") paper, stapled at the left margin. 
Do not bind or enclose in folders. Cover pages on 
two copies should be signed by the Initiator or 
Project Director and by the Transmitter. 

Address 

The address label for most proposals should read: 

Research Analysis and Allocation Staff 
Bureau of Research 
U.S. Office of Education 
Washington, D.C. 20202 

Note.— -If the OE regional office in the applicant's 
area has been staffed with a.n Educational 
Research Adviser, proposals for Small Project 
Research and Research Development Grants 
should be sent there. Otherwise, they should be 
mailed to the above address. When in doubt, con- 
tact the Bureau or the appropriate regional office 
listed in appendix C. 
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Number of Copies 
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Small Pro/ecfs.— Send 20 copies of proposals 
requesting $10,000 or less in Federal support, 
with appropriate copies of abstracts. 

General Pro/ecfs.— Send 30 copies of larger pro- 
posals. The Bureau may request 10 additional 
copies of complex proposals. Besides the abstract 
Inside the cover sheet of each proposal, 20 copies 
of the abstract should be provided separately. 

Reviev' Procedures 

There are no deadlines for general submission, 
but some of the special programs process pro- 
posals only at certain times. Proposals are first 
reviewed by Office of Education staff. Those 
which are within the scope of the Bureau’s legis- 
lative authority and show promise of contributing 
to educational improvement are then reviewed by 
specialists outside the Government. These groups 
make recommendations to the Commissioner as 
a basis for his decision and final approval. While 
-many proposals may be processed within 3 or 4 
_Vnths (Small Projects within 2 months), larger 
or more complicated ones may require more time. 

Communications Regarding Proposals 

The initiator will be sent the name of the Bureau 
of Research unit, the project officer, and the 
official identification number assigned to his pro- 
posal. The Office does not give further informa- 
tion about the status of the proposa’ during the 
review and recommendation process. When the 
proposal has been thoroughly assessed In terms 
of the criteria listed earlier In this booklet and 
within the context of pressure for available 
research funds, the Initiator will be notified 
whether his proposed activity Is to be recom- 
mended for negotiation. If the project Is so 
recommended, the Initiator will be sent any 
information necessary for subsequent action. 

U Contractual Procedures 

Negotiation is that process by which a legal and 
mutually satisfactory agreement (contract or 
grant) Is arranged between the applicant and the 
Office of Education. This process transcends a 
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mere final examination of estimated costs. The 
contract specialist works with the research 
specialists (applicants and appropriate staff from 
the Bureau of Research) to clarify «nd set forth 
their mutual agreements about the work and 
services to be carried out and the support to be 
provided. The resulting contract or grant Instru- 
ment, which incorporates the final document 
and/or plan of operation, thus states the condi- 
tions under which the award Is made by the 
Office. 

The investigator Is cautioned that no costs should 
be incurred prior to the receipt of a contractual 
document (grant, contract, or notice of award) 
executed by the Contracting Officer on behalf of 
the Government. All contracts are subject to 
satisfactory fiscal and progress reports. In those 
cases where only an initial phase of a total 
research project has been supported, continuation 
is subject to reappraisal of project aims and 
accomplishments iri/relation to emerging research 
needs and availability of funds In any subsequent 
period. 

Reports 

Reports to be received by project officers Include: 
(1) generally brief monthly and/or quarterly prog- 
ress reports which are intended to facilitate proj- 
ect monitoring; (2) two types of substantive 
reports, (a) Interim reports as appropriate and 
(b) final reports*; and (3) appropriate fiscal 
reports. 



Interim reports include any written material (tex- 
tual, graphic, or tabular) prepared during the 
project period and generally before submission 
of the final report. Often such reports are pre- 
pared for delivery at a professional meeting, for 
distribution to colleagues, or for some other kind 
of presentation of substantive findings. Regard- 
less of the number or variety of progress and 
Interim reports submitted, the project director 
also must submit 15 copies of a comprehensive 
final report. 



* Detailed instructions for preparing these Interim and final 
reports and related information storage and retrieval items are 
provided at the time the project is funded. 
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For projects that will produce nonprint materials, 
the project officer should specify at the time of 
project negotiation the number of copies of such 
materials to be received by the Office of Educa- 
tion, taking into account loan uses by program 
staff, possible uses in regional laboratories, and 
other needs. 

Special Conditions 

Release of Materials 

Materials produced under research contracts or 
grants will be placed in the public domain. When- 
ever any materials not first produced in the 
performance of a particular contract are to be 
incorporated in the new materials, the contract 
must include appropriate arrangements for use 
of materials so incorporated. 

Data-Collection Instruments 

Prior Approval. — Clearance prior to administra- 
tion Is required for all tests, questionnaires, in- 
ventories, interview schedules or guides, rating 
scales, and survey plans used to collect informa- 
tion on identical items from 10 or more individ- 
uals or organizations. This regulation flows from 
the provisions of Federal law, from technical re- 
quirements for proper project-monitoring, and 
from sound considerations of policy in the ad- 
ministration of public funds for educational re- 
search. 



I 

project). Clearance will take all these factors into 
account; thus, questionnaire items that are per- ig 
missibie or desirable in one sample might not jjj 
be so in another. 



Clearance. — ^The clearance process requires six 
copies of each Instrument, with these exceptions: 
Copies need not be submitted of instruments 
which deal solely with cognitive functions or tech- 
nical proficiency (e.g., scholastic aptitude, school 
achievement, vocational proficiency); routine 
demographic information (e.g., age, sex, race, 
residence, school attendance); or routine institu- 
tional information. However, the information 
specified in the preceding paragraph is required 
for ail instruments, whether copies of the instru- 
ment are submitted or not. 

In the process of clearance, attention will be 
given especially but not exclusively to such mat- 
ters as the unnecessary or offensive intrusion of 
inquiries regarding religion, sex, and politics; the 
extraction of self-demeaning or self-incriminating 
disclosures; and the apparent countenancing 
antisocial or immoral behavior. When the re- 
spondents include students below college age, 
the assurance of Informed parental consent will 
be required if unduly sensitive questions are In- 
volved. 

Those who will need to clear data-gathering in- 
struments should request separate, more detailed 
instructions from the Bureau. 
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Description. — All projects involving data-gather- 
ing instruments require submission of a statement 
giving: The exact title of the instrument; the 
purpose(s) of the study and the relation of 
the data-gathering instruments thereto; the na- 
ture end size of the sample. Including the meth- 
od(s) of sampling; the locale of the study; pro- 
visions for anonymity and confidentiality of re- 
sponse; a brief indication of the nature and extent 
of statistical analysis of the data; the estimated 
average time required of the respondent; and a 
single figure giving the estimated cost of the 
survey-proper (as distinguished from the total 



Treatment of Animals 

If animals are Involved In research activities sup- 
ported by the Bureau of Research, their care and 
treatment must conform to the principles set forth 
by the Institute of Laboratory Animal Resources 
of the National Academy of Sciences — National 
Research Councii. 

Research Outside the United States 






There are special requirements applicable to re- |l 
search activities outside the United States, includ- U 
ing Department of State approval, 
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initiator 
Institution or agency 



BUDGET WORKSHEET GUIDE FOR RESEARCH ACTIVITIES 

Duration of proposed activity: 

Beginning date: Ending date: 
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Category 


P-mject Cl 
by line ite 

Federal 


ost 

im* 

Local 


1. DIRECT COSTS 

A Personnel (List ail position titles such as project director, research 
assistant, secretary, consultants,** etc. State percent of time on 
the project, per annum salary, and beginning and ending dates of 
employment for each person.) 






B. Employee Benefits (Summarize benefits such as Social Security, re- 
tirement, etc.) 






C. Travel (Indicate fares and/or mileage at allowable rate; number of 
days per diem,** and rate.) 






D. Supplies and Materials 1. Project Materials 

2. Office Supplies 






E. Communications 






P Services Duplicating and Reproduction 

2. Statistical (Itemize costs) 

3. Testing (Itemize costs) 

4. Other 






G. Final Report Costs 






H. Equipment (Generally, capital equipment may not be purchased with 
research funds. Certain types of equipment may be available from 
Government sources as determined by the Office. If not available, 
cost of rent vs. purchase will be considered.) 






I. Other Direct Costs (List) 






.1. Subtotal, Direct Costs 






II. INDIRECT COSTS (Give basis on which local overhead Is computed, who 
established the rate, when, and period covered.) 






1 ill. TOTAL COSTS 

1 






IV. COST SHARING (Percentage distribution of proposed Federal and local 
support.) 







r ’ ** Consultants: Show rate and number of days under Personnel, transportation and per diem under Travel. 
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APPENDIX B. LEGISLATIVE AUTHORITY FOR RESEARCH AND 
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(in order of enactment) 

P.L 83-480 Agricultural Trade Develop- 
ment and Assistance Act of 
1954 (as amended) 

Sec. 104(k), Research in 

Foreign Countries. 

The Foreign Currency Program, authorized under 
the above law, uses a portion of the Nation’s 
foreign currencies for financing a wide range of 
educational research and related activities carried 
out in certain designated countries abroad. Al- 
most all of these funds are used under agree- 
ments with research organizations in the foreign 
countries themselves, but some may be used by 
U.S. domestic applicants. Countries with P.L. 
480 funds In 1966 were: Burma, Ceylon, India, 
Israel, Pakistan, Poland, United Arab Republic, 
Yugoslavia, Indonesia. 

P.L. 83-531 Cooperative Research Act 
(as amended) 

Sec, 2 (a). Cooperative Re- 
search Program 
Sec. 2 (b). Educational Re- 
search Training 
Programs 

Sec. 4, Educational Re- 
search Facilities 

Inauguration of the Cooperative Research Pro- 
gram in fiscal year 1957 launched the first of the 
systematic research programs to be conducted 
outside the Office of Education with Federal sup- 
port administered by the Office. The 1954 Co- 
operative Research Act authorized the Office to 
enter into jointly financed cooperative arrange- 
ments with universities and colleges and State 
educational agencies for conducting research, 
surveys, and demonstrations In the field of edu- 
cation. The scope of the program was expanded 
by Title IV of the Elementary and Secondary Edu- 
cation Act of 1965 which added support for dis 



semination, construction and operation of facil- 
ities for research and related activities, and 
development of programs to train educational 
researchers. Title IV also expanded eligibility for 
participation. This program permits the Office to 
support a variety of educational research projects 
and programs, thus rounding out a comprehensive 
research effort. 

P.L. 85 864 National Defense Education Act 
1958 (as amended) 

Title VI, Language Development 
Sec. 602, Research and Studies 
Title VII, Research and Experi- 
mentation in More 
Effective Utilization 
of Television, Radio, 
Motion Pictures, and 
Related Media for 
Educational Purposes ^ 
Part A, Research and Experi- 
mentation 

Part B, Dissemination of In- 
formation on New 
Educational Media 

Language Development Research and Studies are 
authorized by Title VI of the National Defense 
Education Act of 1958, which provides support 
for: studies and surveys to determine the need for 
increased or improved instruction In modern 
foreign languages and other fields necessary for 
a full understanding of the areas, regions, or 
countries where the languages are commonly 
used; research on more effective methods of 
teaching such languages and such other fields; 
and development of specialized materials for use 
in training students and teachers in these lan- 
guages or fields. 



Media Research and Dissemination, authorized by 
Title VII A and B of the National Defense Educa- 
tion Act, provides support for (A) research and 
experimentation In the development and evak- 
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ation of activities involving television, radio, mo- 
tion pictures, printed and published materials, 
and related media of communication for educa- 
tional purposes, and (B) the dissemination of in- 
formation concerning new educational media 
through studies and surveys, demonstrations, and 
publications and reports. 

P.L 88-164 Mental Retardation Facilities 
and Community Mental Health 
Centers Construction Act of 1963 
(as amended) 

Sec. 302, Research and Dem- 
onstration Projects 
in the Education of 
Handicapped Children 
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The authorization for Education of Handicapped 
Children Research and Demonstration provides 
support for activities relating to education for the 
mentally retarded, hard of hearing, deaf, speech 
-Impaired, visually handicapped, seriously emo- 
yonally disturbed, crippled, or other health- 
impaired children who, by reason thereof, require 
special education. 

P.L. 88-210 Vocational Education Act of 1963 
Sec. 4(c), Vocational and 

Technical Education 
Research, Develop- 
ment, Training 
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This provision for Vocational and Technical Edu- 
cation Research, Development, and Training pro- 
vides assistance in paying the costs of research 
and training activities and of experimental or de- 
velopmental programs and projects to help meet 
the special vocational education needs of youth 
(particularly youth in economically depressed 
communities who have academic, socioeconomic, 
or other handicaps that prevent them from suc- 
ceeding in regular vocational education pro- 
grams). 

P.L. 89-329 Higher Education Act of 1965 
Title II, Part B, Library Re- 
search and Develop- 
ment 

The authorization of Library Research and De- 
velopment provides support for research and 
demonstration projects to improve libraries and 
training in librarianship; to develop new tech- 
niques, systems, and equipment for handling and 
distributing information; and to disseminate in- 
formation derived from such research and demon- 
strations. 
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Regional Office of Education staff are located at the following addresses: 



Region States Served Address 

Region I Connecticut, Maine, Massachusetts, New Office of Education/DHEW 

Hampshire, Rhode Island, Vermont John Fitzgerald Kennedy Federal 

Building 

Boston, Massachusetts 02203 



Region II Delaware, New Jersey, New York, Penn- 
sylvania 

Region III Kentucky, Maryland, North Carolina, 
Puerto Rico, Virginia, Virgin Islands, 
West Virginia, District of Columbia 

Region IV Alabama, Florida, Georgia, Mississippi, 
South Carolina, Tennessee 

Region V Illinois, Indiana, Michigan, Ohio, Wis- 
consin 



Region VI Iowa, Kansas, Minnesota, Missouri, Ne- 
braska, North Dakota, South Dakota 

Region VII Arkansas, Louisiana, New Mexico, Okla- 
homa, Texas 

Region VIII Colorado, Idaho, Montana, Utah, Wyo- 
ming 



Region IX Alaska, Arizona, California, Guam, 
Hawaii, Nevada, Oregon, Washington, 
American Samoa 



Office of Education/DHEW 
42 Broadway 

New York, New York 10004 

Office of Education/DHEW 
220 Seventh Street, N.E. 
Charlottesville, Virginia 22901 

Office of Education/DHEW 
50 Seventh Street, N.E. 

Atlanta, Georgia 30323 

Office of Education/DHEW 
New Post Office Building 
433 West Van Buren Street 
Chicago, Illinois 60607 

Office of Education/DHEW 
601 East 12th Street 
Kansas City, Missouri 64106 

Office of Education/DHEW 
1114 Commerce Street 
Dallas, Texas 75222 

Office of Education/DHEW 
Federal Office Building, Rm. 9017 
19th and Stout Streets 
Denver, Colorado 80202 

Office of Education/DHEW 
Federal Office Building, Rm. 232 
50 Fulton Street 
San Francisco, California 94102 
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APPENDIX D. RESEARCH AND DEVELOPMENT CENTERS 

The twelve R 8* D Centers in operation as of November 1966 are listed below by date of es- 
tablishment. Ten are supported by Cooperative Research Program funds and two by provi- 
sions of the Vocational Education Act. 

Fiscal year , 

established Name of center, location, and area of inquiry 

1964 Learning Research and Development Center, University of Pittsburgh, Pennsylvania 
(Learning research and instructional practices) 

Center for the Advanced Study of Educational Administration, University of Oregon, 
Eugene (School organization and administration in the societal context) 

1965 Center for Research and Development for Learning and Re-Education, University of 
Wisconsin, Madison (Learning efficiency for children and adults) 

Center for Research and Development on Educational Differences, Harvard Univer- 
sity, Cambridge, Massachusetts (Effects of individual and cultural differences on 

the learning process) 

Center for Research and Leadership Development in Vocational and Technical Edu- 
cation, Ohio State University, Columbus (Research and development activities, in- 
cluding operation of ERIC clearinghouse on adult and vocational education) 

Center for Research, Development, and Training in Occupational Education, North 
Carolina State University, Raleigh (Research and development emphasizing south- 
ern needs in adult and vocational education) 

1966 Research and Development Center in Educational Stimulation, University of Geor- 
gia, Athens (Programs of early and continuous stimulation, 3- to 12-year-olds) 

Research and Development Center in Teacher Education, University of Texas, Aus- 
tin (Teacher education) 

Stanford Center for Research and Development in Tea^.hing, Stanford University, 
Palo Alto, California (Theory and practice of teaching and its effects) 

Center for Research and Development in Higher Educatiori, University of Califor- 
nia, Berkeley (Organization, purposes, and outcomes of higher education) 

Center for the Study of the Evaluation of Instructional Programs, University of Cali- 
fornia, Los Angeles (Study of evaluation processes and techniques) 

1967 Center for the Study of Social Organization of Schools and the Learning Process, 
Johns Hopkins University, Baltimore, Maryland (Influence of social and adminis- 
trative organization of schools on learning of students from diverse backgrounds) 
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APPENDIX E. REGIONAL EDUCATIONAL LABORATORIES 

AMn DADTiriDATIMn states 



] 

1 



As of November 1966 the following Regional Educational Laboratories had been established ■ | 
to serve every section of the continental United States, and a feasibility contract had been j 
negotiated to stiidy needs and resources for a laboratory in the Hawaii-Pacific Basin area. 



Appalachia Regional Educational Laboratory, Charleston, West Virginia (West Virginia, the 
Appalachian counties of Virginia, Tennessee, Kentucky, Ohio, and Pennsylvania) 



Center for Urban Education, New York, New York (Metropolitan New York City and some 
neighboring cities, excluding Long Island) (Evolved from R & D center) 



Central Atlantic Regional Educational Laboratory, Inc., Alexandria, Virginia (Washington, D.C., 
and parts of Maryland, Virginia, Delaware, and West Virginia) 



Central Midwestern Regional Educational Laboratory, Inc., St. Ann, Missouri (eastern Mis- 
souri, southern Illinois, and western Tennessee and Kentucky) 



Cooperative Educational Research Laboratory, Inc., Winnetka, Illinois (Inrilar.: , and parts 
Illinois, Michigan, and Wisconsin) 

Eastern Regional Institute for Education, Syracuse, New York (western Pennsylvania ar 



Far West Regional Educational Laboratory, San Francisco, California (northern Californi; 
all of Nevada except Clark County) 

institute for Educational Innovation, Newton, Massachusetts (New England) 



Mid-Continent Regional Educational Laboratory, Inc., Kansas City, Missouri (weste.m Mis- 
souri, central Oklahoma, and parts of Kansas and Nebraska) 

Northwest Regional Educational Laboratory, Portland, Oregon (Alaska, Montana, Oregon, 
Washington, and northern Idaho) 

Regional Educational Laboratory for the Carolines and Virginia, Rougemont, North Caro- 
lina (North Carolina, South Carolina, and southern Virginia) 

Research for Better Schools, Inc., Philadelphia, Pennsylvania (eastern Pennsylvania, much 
of New Jersey and Delaware) 

Rocky Mountain Regional Educational Laboratory, Denver, Colorado (all or portions of Colo- 
rado, Utah, Wyoming, Idaho, Montana, Arizona, Kansas, and Nebraska) 

South Central Region Educational Laboratory Corporation, Little Rock, Arkansas (Arkansas, 
Mississippi, and portions of Loui.'^Jana, Oklahoma, Kansas, and Missouri) 

Southeastern Educational Laboratory, Atlanta, Georgia (Florida, Georgia, and Alabama) 

Southwest Educational Development Corporation, Austin, Texas (eastern and central Texas 
and so* them Louisiana) 




Michigan-Ohio Regional Educational Laboratory, Detroit, Michigan (Michigan and Ohio) 



New York State, excluding New York City) 
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Southwest Regional Educational Laboratory, Englewood, California (southern California, 
southern Nevada, and southwestern Arizona) 

Southwestern Cooperative Educational Laboratory, Albuquerque, New Mexico (all of New 
Mexico and portions of Arizona, i Oklahoma) 

Upper Midwest Regional EducdtiJtu ^iory, Inc., St. Paul, Minnesota (Iowa, Minnesota, 

North Dakota, South Dakota, and pr* f Wisconsin) 



APPENDIX F. ERIC CLEARINGHOUSES 

During fiscal 1966, the first 13 of a network of ERIC clearinghouses were established. They 
will provids in^'^nvBtion on the subject areas listed below. 

ERIC Clearinghouse on Counseling and Guidance, University of Michigan, Ann Arbor, Michigan 
48104 

ERIC Clearinghouse on the Disadvantaged, Yeshiva University, 55 Fifth Avenue, New York, 
New York 1C003 

ERIC Clearinghouse on Educational Administration, University of Oregon, Eugene, Oregon 
97403 

ERIC Clearinghouse on Exceptional Children, Council for Exceptional Children, National Edu- 
cation Association, 1201 16th Street, N.W., Washington, D.C. 20036 

ERIC Clearinghouse on the Teaching of Foreign Languages, Modern Language Association of 
America, 4 Washington Place, New York, New York 10003 

ERIC ClearinghoL'se on Junior Colleges, University of Califoi. 405 Hilgard Ave., Los An- 
geles, California 90024 

ERIC Clearinghouse on Linguistics and the Uncommonly Taught Languages, Center for Applied 
Linguistics, 1755 Massachusetts Avenue, N.W., Washington, D.C. 20036 

ERIC Clearinghouse on Reading, Indiana University, 204 Pine Hall, Bloomington, Indiana 
47401 

ERIC Clearinghouse on School Personnel, City University of New York, 33 West 42d Street, 
New York, New York 10036 

ERIC Clearinghouse on Science Education, Ohio State Unive'^sity, 1314 Kinnear Rd,, Colum- 
bus, Ohio 43212 

ERIC Clearinghouse on Small Schools and Rural Compensatory Education, New Mexico State 
University, University Park, New Mexico 88070 

ERIC Clearinghouse on Vocational and Technical Education, Ohio State University, 980 Kin- 
near Rd., Columbus, Ohio 43212 

library for Adult and Continuing Education, Syracuse University, 107 Roney Lane, Syracuse, 
New York 13110 
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FOR HANDICAPPED CHILDREN AND YOUTH 



As of the end of fiscal 1966, Instructional Material Centers had been established at the fol- 
lowing locations to serve the educational needs of handicapped children and youth. 

American Printing House for the Blind, 1839 Frankfort Avenue, Louisville, Kentucky 
Colorado State College, Greeley 

Department of Special Education, Superintendent of Public Instruction, 302 State Office Build- 
ing, Springfield, Illinois 

Michigan State University, East Lansing 

The University of Kentucky, Lexington 

The University of Oregon, Eugene 

The University of South Florida, Tampa 

The University of Southern California, Los Angeles 

The University of Texas, Austin 

The University of Wisconsin, Madison 
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March 1967 
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WHERE TO APPLY 



PgopQtale From 



Send To 



DISTRICT OF COLUMBIA 

mXlKKY 

jamum 

NORTH CMOIIHA 
PUERTO RICO 
VIRGINIA 
VIRGIN ISLANDS 
WEST VIRGINIA 



Regional Research Program 
U.S. Office of Bducatlon/i^DEN 
Regional Office III 
220 Seventh S treaty KB« 
Charlottesville, Virginia 22901 



ALABAMA 

FIDRXDA 

GEORGIA 

MISSISSIPPI 

SOUTH CAROLINA 

TENNESSEE 



ILLUtOIS 

INDIANA 

MICHIGAN 

OHIO 

WISCONSIN 



ARKANSAS 
LbUCSIANA 
NEW HEKXCO 
mAHom 

TEKAS 



ALASKA 

ARIZONA 

CAUFORNIA 

GUAM 

HAI^AIl 

NEVADA 

OREGON 

WASHINGTON 

AhERlCAN SAMOA 



Regional Research Program 
U.S* Office of Educatlon/DHEW 
Regional Office IV 
50 Seventh Street N£. Bm« 404 
Atlanta, Georgia 30323 



Regional Research Program 
U.S. Office of Education/IHEW 
Regional Office V 
433 W. Van Buren Street 
Chicago, Illinois 60607 



Regional Research Program 
U.S. Office of Educatlon/DHB^ 
Regional Office VII 
1114 Commerce Street 
Dallas, Texas 75222 



Regional Research Program 
U.S. Office of Educvtlon/DHEW 
Regional Office IX 
Federal Office Building, Bm. 232 
50 Fulton Street 

San Francisco, California 94102 



NOTE: Until educational advisers are appointed for other regions, 
proposals from all other States should be sent to; 

Research Analysis and Allocation Staff 
Bureau of Research 
UcS. Office of Education 
Washington, D*C. 20202 



ERIC 
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Regional Research Program 
Bureau of Research 



Introduc*,tlon 



The Office of Education's Bureau of Research seeks to loq>row. edu- 
cation through support of a variety of educational research and related 
activities conducted outside the Office with authorizations from the 
Congress* Supported activities range from small projects (under $10,000) 
to such comprehensive undertakings as the regional laboratories.* The 
special provision for Small Project Research Is designed to extend oppor- 
tunities for participation In systematic educational improvement to a 
greater numher of individuals and Insvitutions* 

Beginning In the summer of 1966, the administrative responsibility 
for Small Project Research was decentralized to the Office of Education's 
Regional Offices, where small project proposals will be received and 
processed* A list of States served by these offices is shown on page one* 
This decentralisation provides equitable geographic distribution of funds 
allocated to small projects and facilitates the evaluation of proposed 
activities in terms of regional needs and other regional educational 
Improvement efforts* 



What Small Project Research Is 



Definition 

To qualify as a Small Project, a proposal for educational research 
or related activity must meet two basic requirements: 

1. The total loves^ient by the Office of Education is $10,000 or 
less* 

2* The project is scheduled for completion within a period of 13 
months* 

Proposals requesting more support or requiring more time shour.d be 
submitted through other Bureau programs* 

Purpose 

The specific purpose of this program is to be more responsive to the 
regional needs of the educational comsnmity by: 

1* Providing an opportunity for small projects to receive prompt 
consideration* 



ft 



Those Interested in other kinds of Bureau-supported programs or projects 
should consult the appropriate bulletins available from the Bureau of 
Research* 
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2. Encouragliig small colleges to undertake research programs so that 
their personnel may gain experience in research and related 
activities* 

3« Supporting significant, small-scale educational research projects 
by doctoral and post-doctoral students and felloirs, particularly 
those in developing Institutions. Funding la out of whichever 
legislative authcrisatiou is most appropriate for the particular 
kind of activity In question. The determination of this is made 
by the Bureau of Besearch staff. 

How the Small Project Program Operates 

The over-a3.1 application instructions of the Bureau of Research 
(Appendix A) are generally applicable to requests for support of small 
projects. For exaxopXe, eligible parties for small projects are the same 
as for other kinds o£ activities supported through the Bureau, and the pro- 
vision for progress, interim, and final reports applies also to small 
projects. 

However, because the general instructions were designed to be uniformly 
applicable to all kinds of project and program support administered by the 
Bureau, the applicant for small project support may find the following items 
helpful in adapting the over-all instructions to accommodate the size and 
focus of the particular activity he proposes to undertake. 

Activities Anoropriate for Small Project Support 

The criteria for determining whether or not an activity is appropriate 
for support by Bureau authorizations are the. same, regardless of the size 
of the project. However, as a matter of policy, small project funds may 
not be used primarily for conducting meetings, conferences, and seminars. 

Criteria for Evaluation of Small Project Proposals 

Criteria for evaluation of small project applications are the same as 
those for other projects and programs administered by the Bureau, except 
that competition is on a regional rather than National basis. In any case, 
the primary criterion is educational significance— a factor which involves 
the project *8 probable impact and its capacity for continuous and effec- 
tive contribution to educational Improvement. 

Review Procedures for Small Project Proposals 

Small project proposals should he sent to appropriate regional offices 
(see page 1) where they are assigned OE identification numbers and pro- 
cessed in much the same manner as other proposals are handled in the Bureau. 
Following staff review, proposals which show promise of contributing to 
educational liq>rovement are reviewed by specialists outside the Covemment. 
Recommendations then go to the Regional Assistant Commissioner as a basis 
for his decision. Processing from receipt of proposal to notification of 
action, is expected to be completed within two months, except in rare 
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Instances when coiq>llcatlon8 arise beyond the control of ttie regional 
office^ The Office does not release Ixifomatlon about the status of a 
proposal until the final disposition Is determined. At thiit tltae, official 
notification of the action to be taken is sent to the pros]>ectlve con- 
tractor or grantee 9 to the Initiator ^ and to the principal Investigator* 
Hotlficetlon of approval Includes the nans of the OE staff nenber ^o 
will provide liaison between the Office and the project staff. 

Contractual Procedures for Small Projecte 

The general contractual proeadures for other activities receiving 
support through the Bureau of Raaearch are also appUcabJa to ama'^.l projects. 
However 9 arrangeiaeuts to handle negotiation and other fiscal natters through 
the Regional Offices are expected to slnpllfy this process. 

Application Procedures 

AppUcetion for support of a small project Is made through aubnlsslon 
of a formal proposal document which follows the standard format outlined In 
Appendix A. However9 the small project proposal is generally shorter; fewer 
copies (20) with one copy of the abstract placed after each title page are 
needed; and processing can usually be conflated In a relatively shorter 
time Is necessary for larger end more comprehensive activities* 

The proposal document la a comnunlcatlon Instrument which will be 
evaluated by the Office of Education staff and field readers selected for 
their research and other experience and general knowledge of the field* 

It will be the only contact the reviewers will have with the initiator and 
bis Ideas. Tha proposal must stand on Its cwn merits; If it does not 
convey the message 9 staff and field readers will not assume m e anin g or 
intent. 

It Is suggested that the Initiator have a person at his Institution 
who Is not close to the problem read the proposal to make certain it com- 
munlcatas clearly* Hany potentially good research proposals fall short of 
approval bacause they are not clearly written. 

The following suggestions nay be useful to the potential Investigator 
In developing the research proposal: 

lo Tie the relevant literature to the objectives of the proposed 
study* The literature does not have to be exhaustive 9 but 
should contain the most pertinent related studies and show an 
awareness of promising current practices. 

2. Do not attempt to attack a problem in a global fashion* In 
other words 9 do not present so many questions for research that 
the study would be unwieldy or could not be completed In a 
rigorous fashion within the time or funds requested* 

3. Use direct language. Present the proposal la a forward maimer 
and be sure that the Ideas are clearly delineated. Avoid 

the use of educational jargon. 
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4« If the project Involves the application of strategy, describe 
the populaticu characteristics, how they will be selected, the 
experiences to which they will be exposed, and how the evalua-» 
tion will be used to detenoine the effectiveness of the 
experimental treatment (s) . If the activity takes some other 
approach, be sure the description of activities and the 
statement of expected outcomes are clear. 

Above all, adapt the format to the activity, cot the other way around. 

The suggested format in Appendix A is a guide, not a prescription. 
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APPENDIX A — GENERAL APPLICATION INSTRUCTIONS 
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Activities Appropriate for Support 

To be eligible for support administered by the Bureau of Research, an 
activity must (1) be research or research-related, (2) show promise of 
isq;>roving education, (3) have general (not purely local) applicability, 
and (4) be directed toward communicable results. 

The term ''research-related'^ is broadly Interpreted to include such 
activities as the development of materials and improvement of instructional 
practices in general and specific areas, and dissemination and itqftlementa- 
tion of the results of research. Local projects must lead to findings 
significant for other settings if they are to be considered eligible for 
support under research authorizations. This applicability or transfer- 
ability to other settings is the factor which, in many instances, determines 
whether a given proposal is eligible for consideration by the Bureau of 
Research. 

The Bureau's authorizations do not Include funds for general operating 
programs or for activities carried on in conjunction with the research which 
are part of the normal activities of project personnel (e.g«, teacher 
salaries in experimen^l classes) . Bureau uupport may be requested for the 
research component of larger projects where the operating costa are funded 
from other sources. In a few instances, support may be provided for those 
aspects of an operational program which must be modified or established 
in the interests of the project. 

With the esceptlon of Small Project Research and Research Development 
Grant activities, there are no establliihed time or cost limitations for 
projects which may be considered for support. However, contractual 
arrangements usually obligate project funds only out cf the current year's 
appropriations. If a proposed activity im^olvee extensive costs or must 
be continued over subisequent years, the proposal must undergo substantial 
additional evaluation to assure sound initial investment. 

Eligible Parties 

A record of noncompliance with the terms of prior re; earch grants or 
contracts (for exaoq>le— failure to submit a final report) by an applicant 
may be sufficient grounds to refuse consideration of a proposal. 

Grants or contracts may be awarded to colleges, universities. State 
departments of education, or to other public or private agencies, organi- 
zations, groups, or Individuals after proposals have been reviewed by 
Office of Education staff and by appropriate non-Government advisory 
personnel and approved by the U.S. Commissioner of Education. With profit- 
making organizations, contracts follow Federal Procurement Regulations. 
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The type of support Instnnnentt l.e., grant, co8*:-reiob«r8eble con- 
tract, or fixed price contract, osy be selected by contract personnel 
during negotiation. If the authorization providing the funds does not 
specify vhich type of support, consideration is given to providing the 
best balance between fie cal control and flexibility of operation. All 
research grants must Include significant coat sharing by the grantee 
institution. Moreover, in some contract research activities, non*Federal 
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Eligibility for larger projects and for the beginning phase of 
longer projects may be limited to groups which are or can be organized so 
as to «ttnsure proper continued professional and fiscal accountability and 
competence • 

Public schools, school districts, and local educational agencies, in 
order to assure appropriate liaison and communication with operating 
school systems, are requested to send three copies of their proposals to 
their State department of education for comment. To be eligible for 
support under the Vocational Education Act of 1963, proposals submitted 
by local educational agencies must contain evidence of approval by an 
authorized official of the State board of vocational education. 



Criteria for Evaluation of Proposals 

Proposals are evaluated and recommended for approval according to 
these criteria^ 

1. Evlucatlonal significance. 

2. Soundness of design, procedure, or operational plan. 

3', Adequacy of personnel and feelXities. 

4. Economic efficiency. 

5. Other specific criteria, as appropriate. 

Evaluation of a proposal* s significance to education requires more 
than consideration of the project itself. It involves attention to the 
breadth of the project’s probable Impact, its relationship to other on- 
going and conqpleted research, and its capacity for contributing to educa- 
tional liqproveaent within the context of total research needs. 

Educational significance, then, is the first of several conditions 
for support; but if a proposal lacks adequate technical quality, 
personnel, or economic efficiency, it will not supported, regardless 
of its significance. In like manner, no matter how technically excellent 
the proposal may be, its support is contingent on its significance to 
education nationally. 

Funds are not available to support all the good proposals submitted 
tc the Bureau of Research; the current approval rate is about one in five. 
To provide a balanced effort toward educationtil improvement within the 
context cf active conq>etition for available Federal research funds, it is 
sometimes necessary to forego support in a well-researched area in order 
to attend to a neglected but critical one. There also are instances when 
difficult choices must be made between one or two relatively large 
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projtcts and a larger nuober of smaller ones* In any caae« selection of 
any given activity for support is based on systematic evaluation of the 
plan set forth in the formal proposal document and determination of 
whether it meets the needs of education* 

The Proposal Document 

^•'•'lination for support is made by submitting J properly executed 
standard application form (available from the Bureau of Research or 
Regional Offices) and the appropriate number of copies of a formal proposal 
which describes the activity or activities, explains their significance, 
identifies key personnel, and estimates costs* 

Since funds are available only for the most promising activities, 
proposals must be assessed competitively (see Criteria for Eval uatio n of 
proposals) • A uniform proposal format has been designed to eccosmodate 
all Bureau of Research projects, regardless of the authorisations under 
which they are funded* Use of a single format gives the applicant freedom 
to concentrate on the particular activity to be undertaken without having 
to choose among application patterns « 

All proposals must include the standard cover page, a one-page 
abstract, the body section, and personnel and budget items* Within this 
framework, each applicant states the case for his activity* 

The guidelines below arc generally applicable, regardless of the 
magnitude of the proposed project, the area of investigation, or the 
authorisation under which the project may be funded, if approved* The 
applicant la expected to make judicious adaptations of this format to 
accosamdate the kind of research or related activity he proposes to 
undertake* 

Cover £tige * —Nothing may precede this page* It contains only the 
following information in the order indicated, and two coplea must be 
signed. (See page 10 of the pamphlet "Office of Education Support for 
Research and Related Activities” for correct format*) 

II* The Abstract occupies a single page, identifies the proposal, and 
concisely and ainply summarizei? the contents* To acconmodate the various 
uses made of this page, the abstract must be written in language under- 
standable by an informed layman* One abstract is placed after the title 
page in each copy of the formal proposal and 20 copies ere provided 
separately. At the top c' the abstract page, the following items should 
appear on separate lines: Title of Project, Principal Investigator, 
Co'&tracting Agency, Amount of Federal Funds Requested, and Proposed 
Beginning and Ending dates* The summary portion of the abstract has three 
parts: A statement of the purposes, objectives, or nature of the project; 
an indication of expected contribution to education; and a compendium of 
procedures or description of what is to bo done* 
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III* The Bod\ the proposal communicates the project director's plan and 
its probable exf ectiveness . It should be clear, concise, forthright, and 
complete, and should be kept within 20 single-spaced c typewritten pages. 
Proposals for exceptionally extensive or complicated projects may subse- 
quently be developed into larger documents with the advice and cooperation 
of Office staff. The body of Small Project Research proposals should not 
exceed 10 pages* 

Within the body section, there are normally three parte which should 
be subtitled as appropriate for the particular kind of research, development, 
or dissemination activity to be undertaken. These parts answer the questions; 
Why? What, when, and how? Use of results? Some attention Is also given to 
personnel, facilities, and costs, which are subsequently treated in detail. 

A. Problem and Obleetives .— The first part tells the research or 
related activity should be undertak^’n. It Includes a statement of the 
problem or purposes, review of literature and related research, concise 
statement of objectives, or any other Information necessary to establish a 
sound rationale. 

B. Description of Activities f Procedures) .—This part tells Is 

to be done, when , and It ?.s the basis for determining the degree to 

which the proposed activity can be expected to accomplish the objectives 
or satisfy the need set forth in the first part of the proposal body. It 
delineates procedures, outlines program arrangements, describes materials 
to be produced, or otherwise explains how the activity leads to results 
and how evaluation will be acconq>lished* Allowances for alternatives, if 
any, should be noted. A time schedule for conq>letion of the project is 
usually provided near the end of the proposal. 

The amount and Rind of detailed information tdLll vary according to the 
type of activity* For example, a basic research study— which might include 
rather strict statistical treatment— would he approached quite differently 
from a curriculiim-development activity. A project which is complete within 
itself would not be approached in the same way as a phase of a proposed 
multiphase activity. Preparation of initial materials for a modern foreign 
language would not include the same procedures as the demonstration of new 
media usage or the testing of a pilot study In vocational or technical 
education. 

C* The Use to M Made of Findings . —This part tells how the results 
of the activity may be disseminated and/or Implemented, what contribution 
to education can be expected, what steps should follow. 

IV. Personnel and Facilities . —Personnel and facilities are a major 
determinant of capability. 

Personnel with major responsibilities are listed by name, position, 
title, experience, responsibilities within the project, percentage of time 
comkltted to the activity, and the extent to which this cotcmltment has been 
assured. Consultants who have agreed to serve should be si mil arly identi- 
fied (otherwise, the application should describe the type of consultative 
assistance required). 
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III. The Bod> the proposal commun'lcates the project director’s plan and 
its probable etfectiveness. It should be clea^r, concise » forthright, and 
complete, and should be kept within 20 single-spaced c typewritten pages. 
Proposals for exceptionally extensive or complicated projects may subse- 
quently be developed Into larger documents with the advice and cooperation 
of Office staff. The body of Small Project Research proposals should not 

exceed 10 pages. 

Within the body section, there are normally three parts which should 
be subtitled as appropriate for the particular kind of research, development, 
or dissemination activity to be undertaken. These parts answer the questions: 
Why? What, when, and how? Use of results? Some attention is also given to 
personnel, facilities, and costs, which are sid)scquently treated in detail. 

A. Problem and Objectives .— The first part tells the research or 
related activity should be under taV n. It includes a statement of the 
problem or purposes, review of literature and related research, concise 
statement of objectives, or any other information necessary to establish a 
sound ratioxiale. 

B. Peacription of Activities ( Procedures) .—This part telle is 
to be done, when , and how . It is the basis for determining the degree to 
which the proposed activity can be expected to accomplish the objectives 
or satisfy the need set forth in the first part of the proposal body. It 
delineates procedures, outlines program arrangements, describes materials 
to be produced, or otherwise explains how the activity leads to results 
and how evaluation will be acconqt>li8hed. Allowances for alternatives, if 
any, should be noted. A time schedule for conq>letlon of the project is 
ustndly provided near the end of the proposal. 

The amount and kind of detailed information ^dLll vary according to the 
type of activity. For example, a basic research study— which might include 
rather strict statistical treatment— would be approached quite differently 
from a curriculum-development activity. A project which is complete within 
itself would not be approached in the same way as a phase of a proposed 
multiphase activity. Preparation of initial materials for a modern foreign 
language would not include the same procedures as the demonstration of new 
media usage or the testing of a pilot study In vocational or technical 

education. 

C* The Use to M Made of Findings . —This part tells how the results 
of the activity may be disseminated and/or implemented, what contribution 
to education can be expected, what steps should follow. 

IV. Personnel and Facilities .—Per sonnel and facilities are a major 
determinant of capability. 

Perjonnel with major responsibilities are listed by name, position, 
title, experience, responsibilities within the project, percentage of time 
connitted to the activity, and the extent to which this commitment has been 
assured. Consultants who have agreed to serve should be similarly identi- 
fied (otherwise, the application should describe the type of consultative 
assistance required) . 
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Facilities should be described and the extent to which their use has 
been assured should be indicated. 

V. 7]^ Budget Section of the proposal starts on a new page and the 
tabular presentation shown on page 15 of the paaaphlet “Office of SAw cation 
Support for Research and Related Activities." The applicant dhoald show 
reasonable estimates but be detailed enough to suggest careful enal;c'Si8 

of expected costs and understanding of fiscal responsibilities in connec- 
tion with conducting the proposed activity. Alternative budgets nay show 
allowances for extension of objectives or revision of procedures. Local 
contributlofiy or cost sharing, is required on all grants, encouraged, on 
contracts. Cost sharing may not be avoided by substitution of a contract 
In a situation requiring a grant instrument. Local contribution, while 
cc» necessarily a measure of the project's soundness or potential worth 
and not specifically required on certain contracts, is considered evidence 
of local interest and commitment in the activity. The budget section 
should iucludc 5 &i outline of the sources and amounts of non-Federal support 
and Bay conditions upon which this support is contingent. 

VI. Appended Items 

Other Inf onaat.ion . —Give a brief statement about each of the following: 

(a) If this or a similar proposal has been subuitted elsewhere o give details. 

(b) If this la a proposed extension of, or K.ddition to, a previous or 
current project supported by the Office of ISducatlon, give the Bureau and 
the grant or contract nuniher of the relcced support instruxsentc 

Btevisions . —If this proposal is a resubmission of a previous formal 
proposal, give the Bureau proposal number assigned the origixml proposal 
and describe the major revisions which have been made. (This does not 
apply to preliminary statements submitted for informal review.) 

Report of Other Frolecta .— If any of the primary personnel have a 
current or uncompleted project with the Office of Education or other agency, 
an ^pended statement should Indicate the status of the project, the amount 
of time devoted to it, and the relationships between the current and pro- 
posed projects. 

If any of the personnel have completed a research or development 
project supported by the Office of Education, give information to identify 
it« If findings of the previous project are related to the current pro- 
posal, summarize them briefly. 

Agreement with Cooperating Agencies .—Where agreements with school 
districts or other cooperating agencies are a factor, copies should be 
appended . 

Instruments » —'When an applicant plans to use a published or unpub- 
lished instrument (e.g., a questloxmalre or interview guide) in his study, 
the proposal document should include a copy or, if the instrument is still 
to be developed, a page of sample items and an outline of the complete 
Instrument. (See special conditions, data-collectlon instruments.) 
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Other Items *— »lf necessary » other Items may be added » but appended 
items cannot carry the burden of the request for support* 

Details About Submitting Proposals 

Forms for the standard application should be requested from the Bureau 
of Research at least two weeks prior to the date the copies of the formal 
proposal are to be suo.J.tted* 

Pro^^osals should be on one side of standard (8V’ x 11") paper, stapled 
at the left margin* Do not bind or enclose in folders* Cover pages on 
two copies should be signed* 

Address * 

The address label for most proposals should read: 

Research Analysis and Allocation Staff 
Bureau of Research 
U*S* Office of Education 
Washington, D*C* 20202 

Note*— If the OE Regional Office in the applicant's area has been 
staffed with an Educational Research Adviser, proposals for Small 
Project Research and Research Development Grants should be sent 
there* Otherwise, they should be mailed to the above address* 

When in doubt, contact the Bureau or the appropriate Regional 
Office listed in Appendix B* 

Number of Copies * 

Small Projects *— Send 20 copies of proposals requesting $10,00 or less 
in Federal support, with appropriate copies of abstracts* 

General Projects *— Send 30 copies of larger proposals* The Bureau nay 
request 10 additional copies of complex proposals* Besides the abstract 
inside the cover sheet of each proposal, 20 copies of the abstract should 
be provided separately* 

Review Procedures 

There are no deadlines for general submission, but some of the special 
programs process proposals only at certain times* Proposals are first 
reviewed by Office of Education staff* Those which are within the scope 
of the Bureau's legislative authority and show promise of contributing to 
educational Improvement are then reviewed by apeclaliats outside the 
Government* These groups make reconniendations to the ConuBissioner as a 
basis for his decision and final approval* While many proposals may be 
processed within three or four months (Small Projects within two months), 
larger or more complicated ones may require more tlise* 
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Conmunications Regarding Proposals 

The initiator will be sent the n a m e of the Bureau oi Research *tnlt» 
the project officer » and the official identificatifjn nunber assigned to his 
proposal. The Office does not give further infonoatlon about the status of 
the proposal during the review and reconuendation process. When the pro- 
posal has been thoroughly assessed in terms of the criteria listed earlier 
in this booklet and within the context of pressure for available research 
funds, the initiator will be notified whether his proposed activity is to 
be recoomended for negotiation. If the project is so recommended, the 
initiator will be sent any information necessary for subsequent action. 

Contractua? Procedures 

Negotiation is that process by which a legal and mutually satisfactory 
agreement (contract or grant) is arranged between the applicant and the 
Offlc of Education. This process transcends a mere final examination of 
estifluited costs. The contract specialist works with the research specia- 
lists (applicants and appropriate ^taff from the Bureau of Research) to 
clarify and set forth their mutual agreements about the work and services 
to be carried out and the support to be provided. The resulting contract 
or grant instrument, which incorporates the final document and/or plan 
of operation, thus states the conditions under which the award Is made by 
the Office. 

The investigator is cautioned that no costs should be incurred prior 
to the receipt of a contractual document (grant, contract, or notice of 
award) executed by the Contracting Officer on behalf of the Government. 

All contracts are subject to satisfactory fiscal and progress reports. 

In those cases where only an Initial phase of a total research project 
has been supported, continuation is subject to reappraisal of project 
alms and accomplishments in relation to emerging research needs and 
availability of ftmds in any subsequent period. 

Reports 

Reports to be receive 1 by project officers Include: (1) generally 

brief monthly and/or quarterly progress reports which are intended to 
facilitate project monitoring; (2) two types of substantive reports, 

(a) interim reports as appropriate and (b) final reports;* (3) appropriate 
fiscal reports. 

Interim reports include any written material (textual, graphic, or 
tabular) prepared during the project period and generally before submission 
of the final report. Often such reports are prepared for delivery at a 
professional meeting, for distribution to colleagues, or for some other 



* Detailed instructions for preparing these interim and final reports and 
related Information storage and retrieval items are provided at the time 
the project is funded. 



kind of proseutatlon of substantive findings. Regardless of the nunber 
or variety of progress and interim reports submitted, the project director 
also oust submit 15 copies of a comprehensive final report. 

For projects that vill produce nonprint matarlals» the project officer 
should specify at the time of project negotiation the number of copies 
of such materials to be received by the Office of Education, taking into 
account loan uses by program staff, possible uses in regtonul laberateriss, 
and other needs. 

Special Conditions 

Release of Materials . 

Materials produced under research contracts or grants will be placed in 
the public domain. Whenever any materials not first produced in the per* 
foxmance of a particular contract are to be incorporated in the nev materials 
the contract must include appropriate arrangements for use of materials so 
incorporated. 

Date-Collection Instruments . 

^Xior Approval . —Clearance prior to administration is required for all 
tests, questionnaires. Inventories, interview schedules or guides, rating 
scales, and survey plans used to collect information on Identical items 
from 10 or more individuals or organisations. This regulation flows from 
the provisions of Federal law, from technical requirements for proper 
project-monitoring, and from sound considerations of policy in the admini- 
stration of public funds for educational research. 

Deacriptlon . —All projects involving data-gathering instruments require 
the submission of a statement reporting briefly: the exact title of the 
instrument; the purpose(s) of the study and the relation of the data- 
g.<ithering instruments thereto; the nature and size of the sample, including 
the method(s) of sampling; the locale of the study; provisions for anony- 
mity and confidentiality of response; a brief indication of the nature and 
extent of statistical analysis of the data; the estimated average time 
required of the respondent; and e single figure giving the estimated cost 
of the survey-proper (as distinguished from the total project) • Clearance 
will take all these factors into account; thus, questionnaire-items that 
are permissible or desirable in one sample mlgh.t not be so in another. 

Clearance . —The clearance process requires six copies of each instru- 
ment, with these exceptions: Copies need not be submitted of Instruments 
which deal solely with cognitive functions or technical proficiency (e.g., 
scholastic aptitude, school achievement, vocational proficiency) ; routine 
demographic Information (e.g., age, sex, race, residence, school attendance); 




or routine institutional infonnatlon. However, the information specified 
in the preceding paragraph ie required for all instruments, whether copies 
of the instrument are submitted or not* 

In the process of clearance, attention will be given especially but 
not exclusively to such matters as the uzmecessary or offensive Intrusion 
'of inquiries regarding religion, sex, and politics; the extraction of 
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nanclng of antisocial or immoral behavior* When the respondents include 
students below college age, the assurance of informed parental consent will 
be required if unduly sensitive questions are involved* 

Those who will need to clear data-gathering instruments shoold request 
separate, more detailed instructions from the Bureau* 

Treatanent of Animals 

If animals are Involved in research activities supported by the Bureau 
of Research, their care and treatment must conform to the principles set 
forth by the Institute of Laboratory Animal Resources of the National 
Academy of Sciences*- National Research Council* 

Research Outside the United States 

There are special requirements applicable to research activities 
outside the United States, including Department of Stats approval* 




APPENDIX B— OE REGI(»?AL OFFICES 

BegLonal Office of Education Staff ate located at the following addresses: 



Region 
Region X 

Region II 
Region III 

Region XV 
Region V 

Region VI 
Region VII 
Region VIII 

Region IX 



States Served 



Connecticut* Maine* Massachusetts* 
New Hampshire* Rhode Island, 
Vermont 



Delaware, New Jersey, 
New York, Pennsylvania 



Kentucky f Maryland, North 
Carolina, Puerto Rico, Virginia, 
Virgin Islands, Vest Virginia, 
District of Columbia 

Alabama, Florida, Georgia, 
Mississippi, South Carolina, 
Tennessee 

Illinois, Indiana, Michigan, 
Ohio, Wisconsin 



Iowa, Kansas, Minnesota, Missouri, 
Nebraska, North Dakota, South 
Dakota 

Arkansas, Louisiana, New Mexico, 
Oklahoma, Texas 



Colorado, Idaho, Montana, 
Utah, Wyoming 



Alaska, Arizona, California, 
Guam, Hawaii, Nevada, Oregon, 
Washington, American Samoa 



Addriss 



Office of Education/DBEW 
John Fitzgerald Kennedy Federal 
Building 

Boston, Massachusetts 02203 

Office of Education/DHEW 
42* Broadway 

New York, New York 10004 

Office of Education/DHEW 
220 Seventh Street, N.E, 
Charlottesville, Virginia 22901 



Office of Education/DHEW 
50 Seventh Street, N.E. 

Atlanta, Georgia 30323 

Office of Education/DHEW 
New Post Office Building 
433 West Van Buren Street 
Chicago, Illinois 606Q7 

Office of Education/DHEW 
601 East 12th Street 
Kansas City, Missouri 64106 

Office of Educatlon/IBEW 
1114 Commerce Street 
Dallas, Texas 75222 

Office of Education/DHEW 
Federal Office Build/ )g, Rm. 9017 
19th and Stout Stree;:8 
Denver, Colorado 80202 

Office of Education/DHEW 
Federal Office Building, Rm. 232 
50 Fulton Street 
San Francisco, California 94102 



